How to Remove a Solaris Patch While Booted From a Network or CD-ROM
Enda O'Connor, April 2009
This article covers the following topics:
- Introduction
- Steps for Mounting the Root File System and Other Necessary File Systems
- Mounting Examples
- Running
patchrm
- For More Information
Introduction
This document provides instructions on how to run the patchrm
command in the Solaris Operating System booted from a network or CD-ROM against a non-live system that has been mounted from such media. There are instructions for mounting the root file system and any other required file systems under the /a
mount point after the system is booted from the media.
The main reason for booting alternate media in order to run patchrm
is when the live system has been rendered unbootable and proper root-cause analysis has led to the decision to use patchrm
on one or more patches.
Caution: Before running patchrm
, it is vital that you investigate and eliminate all other potential causes of the problem. Only then, based on sound analysis, should you run patchrm
. Blindly removing patches, especially patches such as Kernel Updates, without properly understanding why the system is no longer functioning correctly, in most cases leads to a worse state than if a proper understanding of the problem has been arrived at. For instance, if a post-patch script that installs a new bootblock fails due to insufficient space in /platform
and the system no longer boots, running patchrm
will most likely only exacerbate the underlying issues (insufficient space), whereas if the problem is understood, unwanted or unnecessary files can be removed, a proper bootblock can be installed, and the system can be booted.
So the first step must be to determine the root cause of the problem that led to an unbootable or corrupted system to begin with. This step is vital; without doing this, patchrm
will most likely only:
- Render the system unrecoverable (from a situation where recovery would most likely have been possible)
- Render root-cause analysis impossible, leading to the situation just reoccurring again
- Most importantly, not resolve the underlying problem, leading to a reoccurrence in the future
To facilitate proper root-cause analysis, please read the BigAdmin article Analyzing a patchadd or patchrm Failure in the Solaris OS, which describes what files are most likely to be relevant for determining the root cause of a patch-related failure. When used with this document, the Patch Analysis article allows you to gather important system information that should help identify the underlying issue that caused the failure.
Only as a last resort should you remove a patch without understanding the reason why it needs to be removed; you need to understand more than just that the patch appears to have caused a problem.
A recurring theme during software maintenance is that a system fails to reboot, but this problem can be caused by numerous issues, such as underlying disk instability resulting in corruption. Removing a patch based on this, will make the situation much worse in all likelihood.
The recommended way to prevent problems with critical software, such as the operating system, is by using mirroring on the root file system, particularly by using Solaris Volume Manager. This method allows you to mirror the operating system on two physical disks, and prior to patching, you can break the mirror and then patch only one half of the mirror. If a problem is discovered, it should be possible to boot from the other half of the mirror (the unpatched half).
It is strongly advised that you use Solaris Volume Manager for mirroring root file systems, as opposed to using Veritas Volume Manager (VxVM) mirroring of root file systems. This is due to VxVM disk encapsulation, which, depending on the disk layout and free partitions on the root disk, might create a hard-to-manage disk layout that causes issues when trying to rescue such a layout.
Also, there is the common misconception in the sys admin world that encapsulating a disk in VxVM equates to mirroring. This is a major mistake. Encapsulation is only the first step towards mirroring a root file system, whereby the currently installed root file system disk is given over to VxVM control and the data is preserved (encapsulated). You then need to mirror the encapsulated root disk using further VxVM commands.
Please read the BigAdmin article How to Split a Root Mirrored With Solaris Volume Manager Prior to Updating Software for information on how to properly break a Solaris Volume Manager mirror and later reattach the mirror.
If your system's boot disks are managed by VxVM, it is strongly advised that you read the Sun BluePrints document Towards a Reference Configuration for VxVM Managed Boot Disks (pdf, August 2000), so you are aware of the issues that affect boot disks that are managed by VxVM.
Steps for Mounting the Root File System and Other Necessary File Systems
- Boot from the media or the network.
- Mount the root file system and any other required file systems. This process is covered in the next section, Mounting Examples.
Mounting Examples
Example 1: Using Standard UFS Disk Slices
Assume that /
, /var
, and /usr
are on separate disk slices with non-global zones mounted under the /zones
disk slice. In this case, you would do the following:
# mount /dev/dsk/c0t0d0s0 /a
# mount /dev/dsk/c0t0d0s4 /a/var
# mount /dev/dsk/c0t0d0s5 /a/zones
# mount /dev/dsk/c0t0d0s6 /a/usr
The root file system is now mounted under /a
.
Now run fsck
to check the file systems:
#/usr/sbin/fsck -F ufs -m /dev/dsk/c0t0d0s0
The -m
flag instructs fsck
to check, but not repair, the file systems.
If a repair is necessary, run the following:
/usr/sbin/fsck -F ufs -o p /dev/dsk/c0t0d0s0
Make sure you retain the output from the previous commands in case it becomes necessary to run fsck
to actually repair the file system.
For this example, you must be booted from at least the Solaris 10 10/08 OS media.
Issue the following command, which displays the list of pools that are available for importing:
zpool import
After the root pool has been identified, run the following:
zpool import -R /a <pool_name>
If there are non-global zones on the system and they reside on a pool, import the pool as well using zpool import -R /a/zones <pool_name>
, assuming the pool is mounted on /zones
, as in the first example.
If there are non-global zones installed and the zonepath resides on a Solaris Volume Manager metadevice, for example, if the following configuration existed prior to patching:
/zones on d20
root@oyster # metastat -p d20
d20 -m d21 d22 1
d21 1 1 c0t2d0s0
d22 1 1 c0t3d0s0
Then mount /zones
( /dev/md/dsk/d20
) under /a
:
# cp /a/kernel/drv/md.conf /kernel/drv/md.conf
Now update the Solaris Volume Manager driver to load the configuration:
# update_drv -f md
Ignore any error messages from update_drv
:
# metainit -r
If you have mirrors, you should run metasync
to get them synced at this point. Use metastat
to see if a sync is required or has completed. After the sync has completed, you can proceed to mount the metadevices:
# mount /dev/md/dsk/d20 /a/zones
Please see Example 3: Mounting When the System Is Running a Solaris Volume Manager Mirror, which describes how to run fsck
on any metadevices that were mounted.
Example 3: Mounting When the System is Running a Solaris Volume Manager Mirror
In this scenario, the system was running a Solaris Volume Manager mirror prior to the system becoming unbootable. In such a case, you need to do some manual work to enable you to mount the /dev/md/dsk/*
metadevices in order to run patchrm
.
So, assume the root disk is laid out as follows:
/
ond10
/var
ond20
Run the following:
root@oyster # metastat -p d10
d10 -m d11 d12 1
d11 1 1 c0t2d0s0
d12 1 1 c0t3d0s0
root@oyster # metastat -p d20
d20 -m d21 d22 1
d21 1 1 c0t2d0s1
d22 1 1 c0t3d0s1
First, boot from the media:
#boot net -s
Now mount one of the subdisks read-only, so you cannot accidentally damage the subdisk:
# mount -o ro /dev/dsk/c0t0d0s0 /a
Then set up the current booted environment so it can use Solaris Volume Manager:
# cp /a/kernel/drv/md.conf /kernel/drv/md.conf
# umount /a
Now update the Solaris Volume Manager driver to load the configuration:
# update_drv -f md
Ignore any error messages from update_drv
:
# metainit -r
If you have mirrors, you should run metasync
to get them synced at this point, if a sync is required. Use metastat
to see if a sync is required or has completed. After the sync has completed, you can proceed to mounting the metadevices:
# mount /dev/md/dsk/d10 /a
# mount /dev/md/dsk/d20 /a/var
If non-global zones are installed, they need to be mounted prior to running patchrm
. Please see Example 1: Using Standard UFS Disk Slices and Example 2: Using ZFS Root for details on how to mount any non-global zones.
Run fsck
to check the file systems:
#/usr/sbin/fsck -F ufs -m /dev/md/dsk/d10
The -m
flag instructs fsck
to check, but not repair, the file systems. If a repair is necessary, run the following:
/usr/sbin/fsck -F ufs -o p /dev/md/dsk/d10
Make sure you retain the output from the previous commands in case it becomes necessary to run fsck
to actually repair the file system.
Running patchrm
After all the necessary file systems have been mounted, the next step is to do a dry run by running the following:
#patchrm -a -R /a <patch_number>
This command does not update any of the installed software, but it validates that the necessary file systems are mounted, especially if there are non-global zones installed.
If this command fails, you must resolve any issues prior to running patchrm
. Make sure all file systems that contain non-global zones have been mounted, and then try again.
After patchrm -a -R /a
passes, run the following:
#patchrm -R /a <patch_number>
Take care to record the exact output for any future debugging.
Note: Do not use any chroot
commands when running patchrm
from media. Also, do not run patchrm
from the mounted system, for instance:
# /a/usr/sbin/patchrm
For More Information
Here are some additional resources:
- Sun download site
- Sun training courses at https://www.oracle.com/sun/
- Forums, such as Sun forums and the BigAdmin Discussions collection
- Product documentation at https://docs.oracle.com/en/ and the Documentation Center
- Support:
- Sun resources:
- Community system administration experts
- Events of interest to users of Sun products:
November 2009: Script was extended to collect additional data for detailed patch and package analysis.