Patch Management Best Practices

Enda O'Connor, April 2008

Introduction

Patch management is a quite complex and involved topic. Unfortunately, there is no one patch management plan that fits all situations. Everyone must determine the plan that fits best with their own environment and business objectives. A plan should be reviewed periodically because the environment and business objectives change over time, new tools and practices evolve, and operating systems evolve. All of these changes require modifications to existing patch management plans.

This article covers the following topics:

Patch Content

People most often associate patches with delivering bug fixes only, and while this might have been true in early Solaris releases, this has progressively ceased to be the case. With the advent of the Solaris 10 Operating System, patches are used for the following purposes:

  • Deliver bug fixes.
  • Deliver new functionality, sometimes quite significant functionality, such as ZFS or NewBoot (GRUB).
  • Deliver new hardware support.
  • Deliver performance enhancements or just enhancements to existing utilities.

Some features require the installation of new packages. These new packages are typically available only by installing or upgrading to a Solaris Update Release that contains the new packages.

But any change to pre-existing code is always delivered in a patch. New functionality, such as ZFS and NewBoot, is delivered entirely by patches, enabling businesses to take advantage of the new functionality without having to upgrade to a newer release of the Solaris OS. So, Sun ships some new functionality in standard patches. Sun also ships new hardware support in patches for much the same reason that Sun ships new functionality, that is, the need to get support for such hardware to market quickly and yet maintain a stable release model going forward.

This document goes through some basic scenarios for creating a patch management strategy for any given organization. Again, it must be stated that the strategies outlined here are only guidelines. Every organization is different in both environment and business objectives. Some organizations have specific guidelines on change management that must be adhered to when developing a patch management plan. Customers can contact Sun Services to help develop an appropriate patch management strategy for their specific circumstances.

The four basic strategies outlined here are as follows:

  • Proactive patch management
  • Reactive patch management
  • Security patch management
  • Installing a new system

We will discuss the tools most appropriate to each strategy, where to locate the patches or patch clusters to apply, and any useful information and tips that are appropriate to a given strategy.

Note : Always make sure you apply the latest revision of the patch utilities patch first before adding any other patches. The patch utilities patch must be applied to the live system in all cases. In the rest of this document, it is always assumed that the latest patch utilities patch has been applied first.

For the Solaris 10 OS, the patch utilities patches are as follows:

  • SPARC: 119254-xx
  • X86: 119255-xx

Note : From the perspective of accessing patches, patches that address security issues remain free. So do patches that provide new hardware drivers. Customers must have a valid support contract to access most other Solaris patches, including the Solaris patch clusters, such as the Recommended Patch Cluster or the Sun Alert Patch Cluster. See Solaris Subscriptions or Sun Spectrum Service Plans for information on how to buy a Solaris Subscription (Solaris OS only) or Sun Spectrum (entire system) support contract. These support contracts entitle customers to access all patches, plus a wide range of additional support services.

Note : It is strongly advised that you read the Special Install Instructions for all patches prior to installing them. In some cases, the Special Install Instructions are updated after the patch has been released to SunSolve, in order to clarify issues surrounding the particular patch installation or to notify users of newly identified issues.

Proactive Patch Management Strategy

The main goal in proactive patch management is to prevent unplanned downtime or, in other words, problem prevention. The idea behind proactive patching is that in most cases, problems that can occur have already been identified and patches have already been released. So, the problem becomes mainly one of identifying important patches from the user perspective and applying them in a safe and reliable manner.

In all cases of proactive patching, it is assumed that the system is already functioning normally. Why patch a system that is functioning normally, since any change implies risk, which in itself implies downtime? As with any system that is functioning normally, there is always the chance that some underlying, known issue can cause a problem. Such issues can include the following:

  • Memory corruption that has not yet caused a problem
  • Data corruption, most of which is silent until such data is read back in
  • Latent security issues
  • Panics due to code paths that have not been exercised before

Security issues are a good example of the value of proactive patching. Most security issues are latent issues, that is, they exist but are not causing issues yet. However, it is important to take proactive action to prevent such issues from being exploited.

In comparison to reactive patching, which is explained in the Reactive Patch Management Strategy section, proactive patching, in general, implies more change as well as regularly scheduled maintenance windows.

Sun strongly recommends that proactive patching be the strategy of choice in those situations where it is applicable. Proactive patching is recommended mainly for the following reasons:

  • Proactive patching reduces unplanned downtime.
  • Proactive patching prevents systems from experiencing known issues.
  • Proactive patching provides the ability to plan ahead and do appropriate testing before deployment.
  • Planned downtime for proactive maintenance is usually much less expensive than unplanned downtime for addressing issues reactively.

Identifying and Downloading Patches That Are Appropriate for Proactive Patching

The recommended way of tracking issues relevant to proactive patching is to register to receive Sun Alerts, which can be done on the SunSolve home page under the Sun Alerts section. This section also provides access to all recent Sun Alerts along with Sun Alert patch reports. It is highly likely that the patches identified will all be contained in the most recent Recommended Patch Cluster.

Note : Both the Recommended and Sun Alert clusters contain only core Solaris OS patches. They do not contain patches for Sun Java Enterprise System, Sun Cluster software, Sun Studio software, or Sun N1, and they do not contain other non-Solaris OS patches that address security, data corruption, or system availability issues.

Core Solaris Tools for Patching

The recommended method of proactively applying patches is to use Solaris Live Upgrade. Solaris Live Upgrade consists of a set of tools that enable users to create an alternate boot environment that is a mirror copy of the current boot partition and then patch the newly created boot partition prior to making it live. Once patched, the new boot partition can be booted. The benefits of using Solaris Live Upgrade are the following:

  • There is decrease in downtime because the only downtime that is needed is the time to boot between the current live partition and the newly patched partition. Patching is not done on the live partition, so the system can continue in production until it is suitable to boot to the newly patched partition.
  • If problems occur, you can boot back to the original partition. No patchrm operation is needed. Just reactivate the original partition and reboot.

If Solaris Live Upgrade is not applicable to the system being patched, it is recommended that the appropriate patches are downloaded and all requirements are identified, and then the patches can be applied using only patchadd.

Solaris Live Upgrade might not be applicable under the following circumstances:

  • There are not enough disk resources available to set up an alternate boot partition. Remember that if you are using Solaris Volume Manager, you need extra resources in order to set up a Solaris Volume Manager alternate boot partition.
  • You are using Veritas Storage Foundation to encapsulate the root disk.

If Solaris Live Upgrade is being used, it can also be used to apply the Recommended Patch Cluster. One caveat is that you must pass the -t option into patchadd using the -O option to luupgrade, for example:

#cd 10_Recommended #luupgrade -t -n be3 -O -t -s . ./patch_order The first -t tells luupgrade that you are installing a patch. The second -t is actually being passed to patchadd through the -O option to luupgrade. It instructs patchadd to skip patch dependency verification.

When using Solaris Live Upgrade is not an option, the Recommended Patch Cluster can be installed using the cluster_install script that comes with the Cluster. The cluster_install script invokes patchadd to apply the patches to the live boot environment in the installation order specified in the patch_order file in the Cluster. If additional patches are to be applied to a Solaris 10 system using patchadd, patchadd -a -M can be useful for identifying any missing requirements and identifying a valid installation order for the patches.

The following is a basic example of running patchadd -a -M:



#patchadd -a -M <dir containing patches>

The -a argument instructs patchadd to do a dry run, so that no software gets installed and no changes are made to the system. The output from the command is verbose but consists of an ordered list of patches that can be installed, plus it clearly identifies any patches that cannot be installed due to dependencies that must be satisfied first. It is strongly recommended that you do not apply patches using -M without the -a option, due to several bugs in the current implementation. (In other words, do not drop the -a option.)

It is recommended that after the complete list of patches has been identified, the patches be installed one by one using just patchadd (without the -M option).

Here is an example of using patchadd in a loop, where patch_order_file is the ordered list from patchadd -a -M:



#patchadd -q -a -M . |grep "Approved patches:" |sort -u \
|sed -e "s/Approved patches://g" > patch_order_file 2>&1

#Cat  patch_order_file
120900-03 121333-04 119254-50

#for i in 'cat  patch_order-file'
do
         patchadd $i
done

In this patchadd -q -a example, -q instructs patchadd to run in "quiet" mode and also output headings for the installable patches, which are called "Approved patches."

While this method of applying patches (using just patchadd as opposed to using Solaris Live Upgrade) has the major disadvantage of requiring you to patch the live system, which increases both downtime and risk, by using the -a option to inspect the patches before applying them against the actual system, you can reduce the risk.

Note : The -M option to patchadd is available only in the Solaris 10 FCS (03/05) release and later.

The information in this section describes how to use the core Solaris Live Upgrade and patch utilities to patch a system. Sun also has a range of higher-level patch automation tools. See the Patch Automation Tools section for more information.

Proactive Patching on Systems With Non-global Zones

On systems where there are non-global zones, patching can still be done using Solaris Live Upgrade. If you are already running the Solaris 08/07 OS, then Solaris Live Upgrade can be used to apply patches without any further issue. If you are running a Solaris 10 release prior to 08/07, you need to add the Solaris 08/07 Solaris Live Upgrade packages to the live system and then apply a list of patches that are required in order to get Solaris Live Upgrade working in an environment that has non-global zones deployed. The Solaris Live Upgrade Software: Minimum Patch Requirements document outlines the required patches and the process that must be followed to get Solaris Live Upgrade from the Solaris 08/07 release working on a pre-08/07 Solaris release.

Note that the list of patches to apply to get Solaris Live Upgrade working in an environment with non-global zones is quite large, and they must be applied to the live running environment. But once this is done, Solaris Live Upgrade can be used to patch going forward, as normal, with all the benefits that Solaris Live Upgrade provides.

If Solaris Live Upgrade is not an acceptable option, then Sun recommends that you use the same method outlined in the Core Solaris Tools for Patching section, that is, identify all the patches required and use patchadd -a -M to identify any missing requirements. As explained earlier, the -a option does a dry run, so no patches will actually get installed.

Pay particular attention to the patchadd -a -M output, and in particular make sure that all non-global zones passed the dependency tests. With non-global zones, the -a option can help identify the following issues:

  • Zones that cannot be booted
  • Zones in which a given patch did not meet all the required dependencies

In such cases, the problem must be identified and rectified before patching can commence.

In the case of systems that have non-global zones configured, it is recommended that you apply patches individually using just patchadd. To facilitate applying multiple patches, you can use patchadd -a -M to produce an ordered list of patches that can be installed individually using patchadd.

Due to current issues with patchadd -M, it is recommended that you do not run patchadd -M without the -a option, because doing so can lead to unrecoverable errors. For the same reason, it is also recommended that you not use luupgrade -t to apply a list of patches using an order file, because it uses patchadd -M internally.

In the case of patchadd, it is also recommended that you first run #patchadd -a to verify that all zones can be booted and that the specified patch is applicable in all zones.

Reactive Patch Management Strategy

Basically, reactive patching occurs in response to an issue that is currently affecting the running system and that needs immediate relief. The most common response to such a situation is usually to apply the latest patch or patches, which might be perceived as being capable of fixing the issue. These patches could be the latest Recommended Patch Cluster or one or more patches that appear to be appropriate. Unfortunately, even though this approach might work if the undiagnosed issue has already been root caused and a patch issued to fix the issue, if the approach does not work, you are often worse off than before you applied the patch or patches.

There are two main reasons why this approach is fundamentally incorrect:

  • The first reason is that even if the problem appears to go away, you don't know whether the patch or patches actually fixed the underlying problem. The patches might have simply changed the system in such a way as to obscure the issue for now, and the problem could recur later.
  • The second reason is that applying patches in a reactive patching session introduces an element of risk. When you are in a reactive patching situation, you must try to minimize risk (change) at all costs. In proactive patching, you can and should have tested the change you are applying. In a reactive situation, if you apply a large number of changes and those changes do not fix the underlying issue (or, even if they do fix the underlying issue), you now have a system issue that still has to be identified as the root cause, and there's a lot more risk involved. Also, there's a greater chance that the changes you applied will have negative consequences elsewhere on the system, which lead to more reactive patching.

So Sun strongly recommends that even when you experience an issue that is affecting the system, you should spend time investigating the root cause of the issue. If a fix can be identified from such investigation, and that fix involves applying one or more patches, then at least the change is minimized to just the patch or set of patches required to fix the problem.

Depending on the severity of the problem, the patch or patches that fix the issue will be installed at one of the following times:

  • Immediately to gain relief
  • At the next regular maintenance window, if the problem is not critical or a workaround exists
  • During an emergency maintenance window that is brought forward to facilitate applying the fix earlier

Identifying Patches for Reactive Patching

Identifying patches that are applicable in a reactive patching scenario can often be complex. In a lot of cases, depending on support, contact with official Sun Support channels will be initiated. But as a starting point, you should do some analysis. Some tools that are useful in starting this analysis might include the following, depending on the issue, though this is not an exhaustive list:

  • The truss command (options such as -fae might let you dig deeper)
  • DTrace
  • Various system analysis tools, such as kstat, iostat, netstat, prstat, sar, vmstat, and even mdb

There is no one, standard way of analyzing an issue because each issue involves different choices. Using debug level logging (when appropriate) and examining various log files might also provide insight into the issue. Also, a proper recording system that records changes to the system should be considered, so that recent system configuration changes can be investigated as possible root causes. When providing data to Sun engineers, using Sun Explorer logs is strongly recommended, because this provides a good foundation on which to start an analysis of the system.

Tools for Applying Patches in a Reactive Patching Scenario

If a fix has been identified and a patch has been downloaded (see the Identifying and Downloading Patches That Are Appropriate for Proactive Patching section for information on downloading patches), Sun recommends that users use Solaris Live Upgrade to apply patches, when possible. If you need to apply the patch or patches immediately (or the issue impacts Solaris Live Upgrade itself), you should first run patchadd with the -a option (since -a does a dry run and does not modify the system). Then inspect the output from patchadd -a prior to actually installing the patch or patches. If more than one patch is being installed, you can use patchadd -a -M to produce an ordered list of patches, which should then be installed individually using just patchadd.

If the system being reactively patched has non-global zones installed, you should apply all patches individually using luupgrade if you are using Solaris Live Upgrade. Otherwise, use just patchadd. At no stage when patching a system with non-global zones should you use the -M option to patchadd, nor should you try to apply a list of patches using an order file and luupgrade. Using Solaris Live Upgrade is covered in more detail in the Core Solaris Tools for Patching section.

In addition to using the core Solaris Live Upgrade and patch utilities to patch a system, Sun also has a range of higher-level patch automation tools. See the section Patch Automation Tools for more information.

Security Patch Management

Security patch management requires a separate strategy, because it requires you to be proactive and yet it requires a sense of urgency. In other words, security fixes deemed relevant might need to be installed proactively before the next scheduled maintenance window.

The recommended method of staying on top of security issues is to register to receive Sun Alerts, which encompasses receiving Security Alerts as well.

Tools for Applying Security Patches

The same rules apply to applying security patches as to proactively or reactively applying patches. The recommended approach is to use Solaris Live Upgrade; otherwise, use patchadd. See the Core Solaris Tools for Patching section for more information.

In addition to using the core Solaris Live Upgrade and patch utilities to patch a system, Sun also has a range of higher-level patch automation tools. See the section Patch Automation Tools for more information.

Installing a New System

Probably the best time to proactively patch a system is while it is being installed. This ensures that when the system boots, it has the latest patches installed and, therefore, avoids any known issues that are outstanding. It also lets you perform testing on the configuration in advance (if testing has been scheduled into the provisioning plan). In addition, it allows you to create a baseline for all installations. Patching during the installation phase requires that you use JumpStart.

The following JumpStart profile examples might be of use:

  • Example 1 (applying an individual patch 119254-50): patch 119254-50 nfs:///export/images/SPARC/10_Recommended
  • Applying the 10_Recommended cluster during JumpStart: patch patch_order nfs:///export/10_Recommended retry 5

Patches can also be applied using finish scripts, as shown in example 4-4 in Creating Finish Scripts.

Patch Automation Tools

Here is a list of automated patching tools:

  • Sun Connection 1.1.1 Satellite Edition:

    • Sun Connection 1.1.1 Satellite Edition is based on Aduva Onstage and Update Connection Enterprise. A central server (Satellite) at the customer site is used to analyze and update all attached client systems in a fully automated manner. It builds upon a central knowledge database fed by Sun. It covers the provisioning of patches, packages, configuration files, and scripts. It is available to customers who pay for it.

    Sun Connection 1.1.1 Satellite Edition provides a solution for customers primarily interested in patch and package provisioning. Sun Connection Satellite is planned to be a component of the recently announced Sun xVM Ops Center 1.0.

  • Patch Check Advanced (PCA):

    • PCA is a popular third party tool developed by Martin Paul. For information on PCA and to download PCA, visit the PCA web site.

    PCA is a robust solution that can meet most customers needs. It provides a reliable and easy-to-use interface, and it can use a local patch server repository to speed up download.

  • smpatch, Update Manager, and Sun Connection - Hosted Edition:

    • The smpatch command-line tool is part of the Solaris OS. It allows customers to analyze and update the Solaris OS with current patches. For customers without a valid support contract, only security and driver patches are available. For customers with a valid support contract, all patches are available.

    Update Manager ( updatemanager) is a GUI wrapper around smpatch and it is also part of the Solaris OS. It can be used to see what patches and updates are available and to easily select the patches that you want to install. Again, for customers without a valid support contract, only security and driver patches are available. For customers with a valid support contract, all patches are available.

    Sun Connection - Hosted Edition is the Internet portal version of updatemanager. You can register all your servers and you can schedule and review the installation of patches from a central portal. This functionality is available only to customers who pay for it.

  • Enterprise Installation Standards (EIS)

    • EIS originated from Sun field personnel who wanted to develop best-practice installation standards for systems installed at customer sites. The EIS patch baseline goes through QA testing prior to release. The images installed by Sun's manufacturers on servers are also based on the EIS patch baseline. Additional testing by Sun's manufacturers plus feedback from the EIS user community raises confidence in the EIS patch baseline further. Since many system installations worldwide use the EIS methodology, any inherent problems quickly appear and can be dealt with. In the event that issues occur with the EIS patch baseline, recommendations are communicated to the EIS community.

    The EIS set of patches, which are considered by Sun Field Engineers as best practice to install on a new system, can also be used to patch existing systems to the same patch level. The EIS set of patches is based on the Recommended Patch Cluster for the Solaris OS with additional patches included by Sun Field Engineers for additional products and to address irritating issues that do not meet the criteria for inclusion in the Recommended Patch Cluster.

    The EIS patch baseline covers the Solaris OS and other products, such as Sun Cluster software, SunVTS software, System Service Processor (SSP), System Management Services (SMS), Sun StorageTek QFS software, Sun StorageTek SAM-FS software, and it includes patches that provide firmware updates. Traditionally, EIS has been available only through Sun Services personnel, but it is now available directly to customers through Sun Connection Satellite. This option provides a good way for customers to patch a defined and tested patch baseline. See the Sun Connection blogs for more information.

For More Information

Here are additional resources:

  • Training:
    • Sun Connection Webinar (requires registration)
    • Sun training courses at http://www.sun.com/training/, with modules about the fundamentals of package and patch administration:
      • Make the Transition to the Solaris 10 Operating System (VC-SA-210-S10)
      • Solaris Operating Environment Essentials for System Maintainers (SM-101)
      • System Administration for the Solaris 10 Operating System Part 1 (SA-200-S10)
      • Solaris 10 Features for Experienced Solaris System Administrators (SA-225-S10)
      • Solaris System Administration for Experienced UNIX Administrators (STS-276)
      • Sun Update Connection - Enterprise Technical Essentials (WET-1451)
  • These and other documents at http://docs.sun.com
    • Solaris Patch Management: Recommended Strategy
    • Documents in the Solaris 10 System Administrator Collection, for example, the System Administration Guide: Basic Administration, with a chapter on managing Solaris patches
    • Sun Update Connection - Enterprise document collection
    • Sun Explorer document collection
    • Solaris 10 8/07 Installation Guide: Solaris Live Upgrade and Upgrade Planning
    • Application Packaging Developer's Guide
  • Related sites and resources: