Tips for Hardening an Oracle Linux Server

by Lenz Grimmer and James Morris

Published July 2012

This article provides tips and techniques for hardening an Oracle Linux server.

Introduction

Oracle Linux provides a complete security stack, from network firewall control to access control security policies. While Oracle Linux is designed "secure by default," this article explores a variety of those defaults and administrative approaches that help to minimize vulnerabilities.

Please remember that the strategies discussed here are presented as options to consider rather than definitive rules to apply—system modifications must always be tested for compatibility and confirmed as supported with the intended application stack.

This article covers the following general strategies and practices for hardening Oracle Linux systems:

  • Minimize software and services. Eliminating unnecessary software packages and services minimizes possible avenues of attack.
  • Tighten networking and user access. The network is a prime point of entry for malicious users and applications. Fine-tuning the network configuration, along with all user access points, helps to prevent unauthorized access.
  • Protect applications and data. Setting up devices, mounts, and file systems appropriately (and in some cases using encryption) helps to safeguard applications and data.
  • Implement security features that enforce policies. In some cases, the security policy may dictate additional mechanisms, such as TCP wrappers, Pluggable Authentication Modules (PAM), or the implementation of Security-Enhanced Linux (SELinux).
  • Follow appropriate operational procedures. In addition to maintaining systems' physical security, apply support patches and security updates promptly. Monitor system logs and audit trails, implementing procedures and tools that look for signs of compromise. Also, conduct security evaluations periodically to review security-related practices and procedures.

To limit scope, this article focuses on the first four bullet points above—specifically strategies related to hardening the Oracle Linux operating system. Operational safeguards, such as system management, auditing, and updates, aren't covered here, although they are just as important. (They might be a topic for a future article.)

More About Oracle Linux Security

Over the past few years, Oracle Linux has evolved into a secure enterprise-class operating system that can provide the performance, data integrity, and application uptime necessary for business-critical production environments.

Within Oracle itself, thousands of production systems run Oracle Linux with numerous internal developers using it as a development platform. It is also at the heart of several Oracle engineered systems, including the Oracle Exadata Database Machine, the Oracle Exalytics In-Memory Machine, the Oracle Exalogic Elastic Cloud, and the Oracle Database Appliance.

Oracle On Demand services—which deliver software as a service (SaaS) at a customer's site, via an Oracle data center, or at a partner site—use Oracle Linux at the foundation of its solution architectures. Backed by Oracle support, these mission-critical systems and deployments speak volumes regarding the built-in security and reliability features of the Oracle Linux operating system.

Released under an open source license, Oracle Linux includes the Unbreakable Enterprise Kernel that brings to market the latest Linux innovations while offering tested performance and stability. Oracle has been a key participant in the Linux community, contributing code enhancements such as the Oracle Cluster File System and the Btrfs file system. From a security perspective, having roots in open source is a significant plus—the Linux community (including experienced developers and security experts) reviews posted Linux code extensively prior to its testing and release. The open source Linux community has supplied many security improvements over time, including access control lists (ACLs), cryptographic libraries, and trusted utilities.

Minimizing the Software Footprint

When you install Oracle Linux, you can reduce the attack surface by installing only the software packages needed for operation. Software packages are a potential source of setuid programs, network services, and libraries that can potentially be used to gain access illegitimately and compromise a system. Using a pretested Kickstart profile provides consistent and precise control over what's installed, lowering security risk as well as administrative effort by automating installations. Alternatively, Oracle Enterprise Manager Ops Center supports importing OS images and explicit provisioning profiles (see the Oracle Enterprise Manager Ops Center Feature Reference Guide).

On systems on which Oracle Linux is already installed, prune out unneeded RPMs to minimize the software footprint. For example, the X-Windows system isn't needed on most servers and can be uninstalled.

Minimizing Active Services

By default, Oracle Linux is configured a minimal set of services: print services (cupsd and lpd), sendmail, sshd, and xinetd (which launches other internet services). Similar to minimizing software, restricting services to only those needed for the server to deliver application services can help to eliminate potential avenues of attack. One approach (although not always feasible) is to configure one type of service per machine (for example, configure Apache HTTP services on one server, NFS services on another, print services on a third, and so forth). This technique limits exposure if a system is compromised.

Optimally, remove software packages associated with a service if the service is not in use. In some cases, it may not be possible to remove the service because of software dependencies—if this is the case, you can disable any services that can't be removed using the service and chkconfig commands.

For services that are in use, be sure to keep software packages up to date, applying the latest Oracle support patches and security updates. To protect against unauthorized changes, secure the file /etc/services, making sure it is owned by root, modifiable only by root, and links to it cannot be created.

Locking Down Network Services

Because networking services are a common avenue of attack, they require particular attention. A common tactic is to minimize network services launched by xinetd, disabling those that are not needed. It's also possible to set resource limits for xinetd services to thwart potential Denial of Service (DoS) attacks. For example, you can limit the number of connection instances for each service or the connection rate by specifying limits in the configuration file /etc/xinetd.conf. (For resource control options, see the man pages for xinetd and /etc/xinetd.conf.)

Port Scanning

Another common strategy for tightening security is to check what services are running on the system using port scanning utilities. These commands are sometimes used to identify open TCP ports and related services:


						
# netstat -tulp
# lsof -iTCP -sTCP:LISTEN
# nmap -sTU

Caution: Before using the nmap command, check governmental regulations pertaining to port scans.

While system administrators might use port scans to identify and eliminate a possible attack vector, a hacker might use them to identify and exploit open ports. For this reason, certain governments might consider forms of port scanning as unlawful cybercriminal activity.

TCP Wrappers

To protect systems from attack via network services, common administrative practice is to configure TCP wrappers and set up firewalls with Netfilter and IPtables. TCP wrappers mediate between incoming client requests and a requested service, and they control access based on defined rules. By editing the files /etc/hosts.deny and /etc/hosts.allow, you can restrict and permit service access for identified hosts or networks.

One method of using TCP wrappers is to signal intrusion attempts from known malicious sources. For example, if a known malicious host or network attempts to crack a system, you can configure the /etc/hosts.deny file to deny access, at the same time sending a warning message to a log file about the event. For example, this entry in the /etc/hosts.deny file logs attempted access by IP address 192.1.168.73 to a known file:

ALL : 192.1.168.73 : spawn /bin/echo `date` %c %d >> /var/log/alert-admin

Netfilter and IPtables

Oracle Linux ships with a built-in kernel subsystem called Netfilter that acts as a packet filtering firewall. Netfilter performs three operations:

  • Packet filtering
  • Network Address Translation (NAT): Hiding IP addresses behind a public IP address
  • IP masquerading: Altering IP header information for routed packets

Filtering rules (stored in kernel tables for each of these operations) determine whether Netfilter allows packets to be received, dropped, or forwarded. Netfilter applies a chain of rules to every packet. The iptables command is the primary interface for configuring rule chains, or you can use the Firewall Configuration Tool (system-config-securitylevel). Note that modifying the rules files /etc/sysconfig/iptables or ip6tables directly is not recommended. The iptables file contains the criteria to apply to packet filtering decisions, such as the type of protocols to filter, packet sources or destinations, and the target action to be taken (for example, drop, accept, and so on).

The packet filtering service is activated using the service or chkconfig commands. To persist through a reboot, you must save filtering rules by putting IPTABLES_SAVE_ON_STOP in the file /etc/sysconfig/iptables-config or by running the following command:

# /sbin/service iptables save

Netfilter and IPtables can track the connection state of running network services and also use that state as the basis of filtering decisions. Tracking state in this way allows you to configure forwarding for established connections, even for inherently stateless protocols such as UDP. For example, this iptables command sets up a rule to forward packets for an already established connection:

# iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

Netfilter and IPtables support flexible and extensive firewall configuration. Administrators can configure NAT and IP masquerading to protect systems that communicate to external networks and port forwarding to control routing. You can also set rule-based packet logging and define a specific log file in /etc/syslog.conf. Be sure to review the documentation on Netfilter and iptables(8) for more information.

SSH

Administrators commonly use the Secure Shell (ssh) for protected, encrypted communications with other systems. Since ssh is an entry point into the system, it should be disabled if it is not needed. If it is required, administrators can tighten its configuration by editing parameters in /etc/ssh/sshd_config(5). Some settings that help to restrict ssh connections include the following:


						
					PubkeyAuthentication yes
					PasswordAuthentication no
					PermitRootLogin no 
					HostbasedAuthentication no
					StrictModes yes

One technique to restrict remote access via ssh is to configure access to a group or certain users. Add lines to /etc/ssh/sshd_config to permit or deny ssh access:

  • Add a line for AllowUsers or AllowGroups to limit ssh to specific users or groups, respectively.
  • Add a line for DenyUsers or DenyGroups to deny specific users or groups, respectively, from using ssh.

Some other settings that help to protect systems are those that cause the ssh client to time out automatically after inactivity:


						
						ClientAliveInterval 600 
ClientAliveCountMax 0

After making changes to the configuration file, be sure to restart the service.

Configuring Mounts, File Permissions, and Ownerships

Some simple steps can help protect data and the integrity of the installed Oracle Linux operating system. First, use separate disk partitions for operating system and user data (that is, separate partitions for /home, /tmp, /var/tmp, /oracle, and so on). This strategy can prevent a "file system full" issue from impacting operations. Establishing disk quotas can also prevent a user from accidentally or intentionally filling up a file system.

To prevent the operating system files and utilities from being altered if a breach occurs, mount the /usr file system as read-only. When it's time to update operating system RPMs, simply remount /usr as read/write using the -o remount,rw option (remount allows you to change mount flags without taking down the system). After performing the update, don't forget to switch back to read-only mode.

To limit user access on certain non-root local file systems (such as /tmp or removable storage partitions), set the noexec, nosuid, and nodev mount options. The noexec option prevents the execution of binaries (but not scripts), nosuid prevents the setuid bit from taking effect, and nodev prevents the use of device files.

POSIX Access Control Lists (ACLs) provide a richer access control model than the traditional UNIX Discretionary Access Controls (DACs) that set read, write, and execute permissions for the owner, group, and other system users. ACLs can define access rights for more than just a single user or group, specifying rights for programs, processes, files, and directories. You can even set a default ACL on a directory, causing its descendents to inherit the same rights automatically. See the setfacl(1) and getfacl(1) commands for more information on how to manage ACLs. The kernel provides ACL support for ext3 and NFS-exported file systems.

Tightening file permissions and checking ownerships is another step to minimize vulnerabilities. Check for world-writable files and directories, as well as any unowned files, and fix these issues if they exist.

The setuid and setgid bits are sometimes set on executables so that they can perform a task that requires other rights, such as root privileges. However, setuid and setgid executables can be exploited through buffer overrun attacks in which unauthorized code is executed using the rights of the exploited process. For this reason, it is a good idea to examine which programs have setuid or setgid on a system. A find command like this shows setuid programs:

# find / \( -perm -4000 -o -perm -2000 \) -print

If the executable isn't one that's actually used (which might be the case for a number of utilities, for example, /usr/bin/rcp, /bin/ping6, and so on, depending on the system), remove the setuid bit from the executable in question:

# chmod -s /usr/bin/rcp

Managing Users and Authentication

Often it's a simple oversight that can leave a gaping security hole. Check the system for unused and unlocked user accounts on a regular basis, and set passwords on any accounts that aren't protected. Make sure that no non-root user accounts have the user ID of 0.

When you install software that creates a default user account and password, be sure to change the vendor's default password immediately. A centralized user authentication method (such as OpenLDAP or other LDAP implementations) can help to simplify user authentication and management tasks, which might help to lower the risk of unused accounts or accounts with null passwords.

To tell exactly who has performed a privileged administrative action, set up the system so it is not possible to log in directly as root. Instead, all administrators should log in to the system first as a named user and then use the su or sudo commands to perform tasks as root. To prevent users from logging in as root directly, edit the /etc/passwd file, changing the shell from /bin/bash to /sbin/nologin. Modify the /etc/sudoers file using visudo to grant specific users authority to perform administrative tasks.

Oracle Linux supports PAM, which makes it easier to enforce strong user authentication and password policies, including password complexity, length, age, expiration rules. PAM also prevents the use of previous passwords. It can be configured to block user access after too many failed login attempts, after normal working hours, or if too many concurrent sessions are opened. PAM is highly customizable by adding different modules, and you can add external password integrity checkers to test password strength.

Additional Security Features and Tools

Oracle Linux offers additional features and tools to augment the built-in operating system security controls. Whether it makes sense to implement these features depends on security requirements, configuration support, and compatibility with your application stack.

SELinux

Developed initially by the U.S. National Security Agency, SELinux adds additional layers of security beyond the basic UNIX Discretionary Access Controls (DAC) mechanisms. Specifically, SELinux adds functionality to support Mandatory Access Controls (MACs) and Role-Based Access Control (RBAC). SELinux mediates access controls according to vendor-provided policies, enforcing access decisions in the kernel.

By default, SELinux uses a policy called targeted, which isolates targeted processes to an operating domain and other processes to an unconfined domain. Use the sestatus command to show whether SELinux is running, the current mode, and the policy in use:


						
						# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          enforcing
Policy version:                 24
Policy from config file:        targeted

In some classified environments, site security policy may require the use of the SELinux mls policy, which supplies stringent Multi-Level Security (MLS) protection. MLS configurations typically require site and security-specific MAC labeling, which entails extensive customization.

Be sure to confirm support and compatibility of SELinux with the application stack. See the Security-Enhanced Linux User Guide (PDF) for more details.

Linux Containers and Control Groups

Available in Oracle Linux 6 with the Unbreakable Enterprise Kernel, Linux Containers (LXC) provide a way to isolate a group of processes from others on a running Oracle Linux system. Linux Containers are a lightweight operating system virtualization technology built on Linux resource management control group (cgroup) capabilities and resource isolation that is implemented through namespaces. The "Containers on Linux" blog article by Wim Coekaerts introduces LXC functionality. The OTN article How I Used CGroups to Manage System Resources in Oracle Linux 6" explores how cgroups can give administrators fine-grained control over resource allocations, making sure that critical workloads get the system resources they need.

For more information, see the Linux Containers chapter of the "Oracle Linux 6 Administrator's Solutions Guide."

Kernel Security Mechanisms

The Linux kernel features additional security mechanisms:

  • Address Space Layout Randomization (ASLR). By randomly arranging the placement of the base, libraries, heap, and stack in a process's address space, ASLR makes it difficult to predict the memory address of the next instruction. This technique, built into the Linux kernel and controlled by the parameter /proc/sys/kernel/randomize_va_space, can thwart certain types of buffer overflow attacks. (Make sure that this kernel setting is compatible with your application stack.)
  • Data Execution Prevention (DEP). Implemented in the Linux operating system, DEP prevents an application or service from executing code from a non-executable memory region. Hardware-enforced DEP works in conjunction with the NX (Never Execute) bit on compatible CPUs.
  • Position Independent Executables (PIE). The kernel supports PIE technology, which means that executable binaries can be loaded at random memory addresses. To generate binaries that are position-independent, the compiler and linker require specific arguments.

Compiler Protections

The gcc compiler features several buffer overflow protection features. Setting the FORTIFY_SOURCE option causes the compiler to issue a warning when it detects a defect such as a potential buffer overflow. The compiler also includes Stack-Smashing Protection in which the compiler puts a stack canary (a known value) before the stack return pointer to discover whether the stack has been "smashed." Like a canary in a coal mine (used to detect air quality problems), a stack canary detects a stack buffer overflow. The canary value is checked before the return, and if it is invalid, then it's likely that malicious code has overwritten the canary value as well as the return pointer.

Cryptography

Data encryption can help to protect both data at rest as well as data in motion. Data at rest—such as data on media and storage devices—can be at risk because of theft or device loss. Data in motion—such as data transmitted over the local area networks and the internet—can be intercepted or altered, so encrypting transmitted data provides protection. Corporate and governmental regulations (including HIPPA, SOX, and PCI) demand the protection of privacy and personal data, making data encryption an increasingly mandated requirement.

There are several cryptography-related strategies you can apply to protect data on Oracle Linux systems. First, when installing systems and application software, specify digitally signed RPM packages. Set the gpgcheck=1 line in the repository configuration file and import the GPG key from Oracle and other software vendors to make sure that downloaded software packages are signed. You can also install RPMs using the Secure Sockets Layer (SSL) protocol, which uses encryption to enhance communication.

To protect against theft, full-disk encryption is becoming more commonplace, especially to bulk-encrypt laptop storage devices or removable devices such as USB drives. Oracle Linux supports block device encryption using dm-crypt and the Linux Unified Key Setup-on-disk-format (LUKS). These technologies encrypt device partitions so that when a system is off, the data cannot be accessed. When the system boots and the appropriate passphrase is provided, the device is decrypted and its data is accessible.

An alternate approach to protect data on a device is to encrypt a file system using the eCryptfs utilities (available in the package ecryptfs-utils). Unlike dm-crypt, which encrypts block devices, eCryptfs technology performs encryption at the file system–level, and it can be applied to protect individual files or directories. For more information, see ecryptfs-setup-private(1), ecryptfs-mount-private(1), and ecryptfs-umount-private(1).

Processors are evolving to support hardware-based cryptography, making encryption and decryption fast and more efficient. Intel has added an Advanced Encryption Standard New Instructions (AES-NI) engine that provides hardware acceleration for cryptography for certain Intel CPUs (see Intel Advanced Encryption Standard Instructions (AES-NI)). Oracle Linux takes advantage of hardware-accelerated encryption on CPUs that support the AES-NI instruction set, speeding up AES algorithms as well as SHA-1 and RC4 algorithms on 32-bit and 64-bit x86 architectures.

Of course, the Oracle Linux operating system uses encryption to support Virtual Private Networks (VPN) and Secure Shell (ssh) and for password protection. By default, Oracle Linux uses a strong password hashing algorithm (SHA-512) and stores hashed passwords in the /etc/shadow file.

Final Thoughts

System security extends well beyond the hardening of the operating system. It requires an overall security architecture and governance framework—including a well-defined security policy, systematic management procedures, and periodic security evaluations—to ensure confidentiality, integrity, and the availability of systems and data. Components such as external firewall devices and intrusion detection systems also contribute to a full security implementation.

Securing systems and OS hardening is a first step in achieving application availability and data protection. Generally speaking, Oracle Linux is configured out of the box with settings and utilities that make it "secure by default." In addition to these default settings, this article gives system administrators some additional strategies to consider.

For more information about security in the Oracle Linux operating system, see the Security Guide.

See Also

Here are URLs for the resources referenced earlier in this document:

And here are some additional resources:

About the Author

Lenz Grimmer is a member of the Oracle Linux product management team. He has been involved in Linux and Open Source Software since 1995.

James Morris is the Linux kernel security subsystem maintainer. He is the author of sVirt (virtualization security), multi-category security (MCS), and the kernel cryptographic API, and he has contributed to the SELinux, Netfilter, and IPsec projects. He leads the mainline Linux kernel team for Oracle and is based in Sydney, Australia.