How to Migrate a Running Oracle Database to Another System

Using Oracle VM Server for SPARC 2.1

by Orgad Kimchi and Roman Ivanov, February 2012

How to set up an Oracle server and its associated storage array so you can migrate a running Oracle database from one system to another without interruption. Includes an overview of, and instructions for, the live migration procedure.

Introduction

This article describes how to take advantage of the Live Migration capability in Oracle VM Server for SPARC 2.1 (previously called Sun Logical Domains) to migrate a running a single instance of Oracle Database 11g Release 2 to another system without interruption.

Note: Oracle Database 11g Release 2 with Oracle VM Server for SPARC has been certified starting with Oracle Solaris 10 10/08. This paper used Oracle Solaris 10 9/10 to evaluate the Live Migration feature of Oracle VM Server for SPARC 2.1.

This article covers the following topics:

Fundamentals

Understanding these concepts will help you better understand the system configuration we recommend.

Planned And Unplanned Downtime

When discussion high availability, we need to distinguish between planned downtime and unplanned downtime.

Planned downtime is a preventive or corrective maintenance task that interrupts day-to-day system operations. Usually, due to constraints that are imposed by the system's software and hardware design, this downtime cannot be delayed.

Planned downtime should be scheduled to minimally impact the availability of a system, because even a minor modification to subsystems can cause a disruption to availability and affect business needs.

Planned downtime is the result of maintenance events, such as repair, backup, or upgrade operations. Repairs are intended to remove faulty components and restore a system to a functional state. Upgrades are implemented to replace the current hardware or software with newer (or enhanced) versions.

Unplanned downtime occurs as the result of hardware faults and environmental events. It is usually urgent to resolve such downtime immediately. This paper does not address this type of downtime.

Introduction to Oracle VM Server for SPARC

Oracle VM Server for SPARC 2.1 is supported on Oracle's UltraSPARC T2 and UltraSPARC T2 Plus platforms and on Oracle's SPARC T3 and SPARC T4 platforms.

Oracle VM Server for SPARC enables you to create multiple virtual systems on a single physical system. Each virtual system is called a logical domain and runs its own instance of Oracle Solaris 10 or Oracle Solaris 11.

Oracle VM Server for SPARC enables the fine-grained assignment of CPU and memory resources to a workload. This paper describes how to migrate an active domain by using the Live Migration feature. A live migration between two machines minimizes the disruption to running applications and provides a powerful platform for managing production environments.

Introduction to Oracle Database

Oracle Database is the leading database application, and as such, it is frequently used on servers. By using virtualization and consolidation, Oracle Database can now run on selected, certified virtual environments, such as Oracle VM Server for SPARC.

Live Migration Overview

The Oracle VM Server for SPARC 2.1 release introduces the Live Migration feature, which enables an active domain to be migrated to another system while the domain continues to run.

Live Migration uses SSL to encrypt logical domain memory structures as they are securely transferred over the network.

The host where the migration is initiated is called the source machine and the host to which the domain is migrated is called the target machine.

Similarly, once a migration begins, the domain to be migrated is referred to as the domain to be migrated. While the migration is in progress, the domain created on the target machine is referred to as the migrated domain.

Live Migration has a set of requirements and restrictions for CPUs, memory, and I/O that must be satisfied on the source and target machines. See "Live Domain Migration Requirements" in the Oracle VM Server for SPARC 2.1 Release Notes.

Benefits of Live Migration

Using Live Migration offers the following benefits:

Maintenance—As an administrator, you can minimize downtime by using the Oracle VM Server for SPARC domain Live Migration feature. If you must shut down some equipment, use this feature to preserve running applications by moving them to some other server.
Hardware resource—If you see a better way to deploy an application over different servers, use domain Live Migration to move an active domain to a machine with more physical memory, more CPU capacity, or a better I/O subsystem. This change improves application performance by keeping the memory structures warm, because any application that uses memory intensively will keep all memory structures in-place. For example, after a live migration, Oracle Database will have its System Global Area (SGA) and other memory structures laid out the same way as before the migration.
Long computations—You can maximize application availability during the migration, because there is no need to shut down the application while the migration is in process. For example, when migrating applications that perform long computations, such as financial data analysis or scientific computations, you do not need to restart from the beginning or perform any intermediate data save and restore procedures.

Hardware and Software Requirements

Before you begin, be sure your software and hardware meet these requirements.

Oracle Database Requirements

To run an Oracle Database single-instance configuration on Oracle VM Server for SPARC, you must use Oracle Database 11g Release 2 (Oracle Database 11.2.0.1). Later versions might also be supported. See the Oracle Database documentation to verify that a particular version is supported and to check for additional requirements. Also see the My Oracle Support Web site (registration required).

Hardware Configuration

Any server that supports Oracle VM Server for SPARC can be used with Oracle Database. Additional hardware, such as external storage arrays or network switches, might be required based on the selected configuration and deployment variant.

Figure 1 shows a typical hardware configuration and cabling that is compliant with domain Live Migration. This example uses the following hardware:

Two SPARC T4-1 servers from Oracle, each with 64 GB of memory and one SPARC T4 processor.
Eight CPU cores per processor and 8 CPU threads per core. Thus, each system has 64 CPU threads (8x8). One internal disk is used on each server as the system disk of the control domain.
One Sun Storage 2540-M2 storage array from Oracle, with sixteen 300-GB SAS disk drives. The storage array is connected to both servers, and each server is connected to the same array controller (A or B) to which LUNs are mapped. The storage array provides one LUN (LUN 0) that is used as the system disk of the guest domain and a second LUN to store the Oracle Database data (LUN 1).
Two 4-GB FC-AL host bus adapters (HBAs), one for each server. These HBAs are used to connect the storage array to both servers.
Four on-board, 1-GbE network interfaces on each server, which are used for network communications.

Figure 1. Hardware Configuration and Cabling

Running Oracle Database on Oracle VM Server for SPARC

Each guest domain that runs Oracle Database should have a minimum of 16 CPUs (2 CPU cores) and 4 GB of memory.

Each control domain that provides virtual devices (virtual disk and virtual network) to an Oracle Database single-instance guest domain should have a minimum of 8 CPUs (1 CPU core) and 4 GB of memory.

Configuration Details

In the example presented in this document, the source server is configured with the control domain and one guest domain (named ldg1) that hosts the Oracle Database. The control domain, named primary, is also an I/O domain. Furthermore, the control domain is used as a service domain that provides virtual device services (virtual disk and virtual network) to the guest domain.

To act as a service domain, the control domain provides the following virtual device services:

One virtual disk service (primary-vds0) exports physical disks as virtual disks to the guest domain.
One virtual console concentrator service (primary-vcc0) provides access to the virtual console of the guest domain. This service uses the port range 5000–5100 for access to virtual consoles.
One virtual switch service (primary-vsw0) is associated with the primary network interface (nxge0).

Software Installation and Configuration

Follow the steps below to configure your environment for live migration.

Installing the Oracle Solaris OS

Oracle Solaris comes preinstalled on new SPARC servers. For these systems, ensure that the appropriate release of Oracle Solaris is installed and verify that the required patches are present.

To reinstall Oracle Solaris on a SPARC system, refer to the Oracle Solaris 10 documentation or Oracle Solaris 11 documentation. In this paper's example, the operating system is installed on the first internal disk of the server.

After Oracle Solaris is installed, the system can be configured and enabled to use the Oracle VM Server for SPARC 2.1 software.

Installing the Oracle VM Server for SPARC Software

The Oracle VM Server for SPARC software also comes preinstalled on new SPARC servers. For these systems, ensure that the appropriate version of Oracle VM Server for SPARC is installed and verify that any required patches are present.

If Oracle Solaris has been reinstalled on the server, you might have to reinstall the Oracle VM Server for SPARC software. Refer to the Oracle VM downloads page.

The following example shows the actions that are performed on each physical server. Each server already has Oracle Solaris and the required patches installed.

Ensure that the system firmware matches the Oracle VM Server for SPARC release that you plan to install. See "Required Software and Patches" in the Oracle VM Server for SPARC 2.1 Release Notes and "Upgrading the System Firmware" in Oracle VM Server for SPARC 2.1 Administration Guide of the documentation.
Download the Oracle VM Server for SPARC 2.1 software from the Oracle VM downloads page.
Extract the archive.
Install the SUNWldm.v package. See "Installing Oracle VM Server for SPARC Software on a New System" in the Oracle VM Server for SPARC 2.1 Administration Guide.
```
primary# pkgadd -d OVM_Server_SPARC-2_1/Product SUNWldm.v
```

Configuring the Storage Array

The Sun Storage 2540-M2 storage array is connected to the control domain, which exports two LUNs (LUN 0 and LUN 1) as virtual disks to the guest domain.

The guest domain uses the first LUN (LUN 0) as the system disk for the operating system and the second LUN (LUN 1) as a disk to store Oracle Database data files. Figure 2 shows the storage layout of the source and target machines.

Figure 2. Storage Configuration

Configuring the Control Domain

After installing the Oracle VM Server for SPARC software, reconfigure the current system to become the control domain. Perform the following actions on each physical server:

1. Create a virtual console concentrator (vcc) service for use by the virtual network terminal server daemon (vntsd) and as a concentrator for all logical domain consoles.
```
primary# ldm add-vcc port-range=5000-5100 primary-vcc0 primary
```

2. Create a virtual disk server (vds).

primary# ldm add-vds primary-vds0 primary

3. Create a virtual switch service (vsw) to enable networking between virtual network (vnet) devices in logical domains.
```
primary# ldm add-vsw net-dev=igb0 primary-vsw0 primary
```

4. Verify that the services have been created by using the ldm list-services command, as shown in Listing 1.

Listing 1. Verifying Services

primary# ldm list-services primary

VDS
   NAME         VOLUME OPTIONS DEVICE
   primary-vds0
VCC
   NAME         PORT-RANGE
   primary-vcc0 5000-5100
VSW
   NAME         MAC               NET-DEV  DEVICE   MODE
   primary-vsw0 02:04:4f:fb:9f:0d nxge0    switch@0 prog,promisc
VCC
   NAME          LDOM     PORT-RANGE
   primary-vcc0  primary  5000-5100
VSW
   NAME          LDOM    MAC               NET-DEV ID DEVICE    LINKPROP DEFAULT-VLAN-ID PVID VID ...
   primary-vsw0  primary 00:14:4f:f8:28:c2 igb0    0  switch@0  1        1               1500 on  
VDS
   NAME          LDOM      VOLUME  OPTIONS  MPGROUP  DEVICE         
   primary-vds0  primary

Oracle VM Server for SPARC uses SSL to encrypt migration traffic to protect sensitive data from exploitation and to eliminate the requirement for additional hardware and dedicated networks.

The speed of the migration operation increases when the primary domain on the source and target machines has cryptographic units assigned. This increase in speed occurs because the SSL operations can be offloaded to the cryptographic units.

Note: The SPARC T4 CPUs have cryptographic instructions built in, so you do not need to assign them to the CPU.

1. Determine whether you have cryptographic devices in the control domain.
```
primary# ldm list -o crypto primary
```
2. Assign cryptographic resources to the control domain.
```
primary# ldm set-crypto 1 primary
```
3. Assign CPUs and memory to the control domain.
You can reduce the overall migration time by adding more CPUs to the primary domain on the source machine. It is best to have at least eight CPUs assigned to each primary domain. For example, the following commands assign eight CPUs and 4 GB of memory to the control domain.
```
primary# ldm start-reconf primaryprimary# ldm set-vcpu 8 primary
primary# ldm set-memory 4G primary
```
4. Reboot the system.
```
primary# init 6
```
After the system reboots, Oracle VM Server for SPARC is enabled and the system is configured with one domain: the control domain, primary. From the control domain, you can create and configure additional domains.
5. Ensure that the Logical Domains Manager (ldmd) and the Virtual Network Terminal Server (vntsd) services are enabled.
```
primary# svcadm enable ldmdprimary# svcadm enable vntsd
```
6. Add a logical domain machine configuration to the service processor (SP). For example, the following command adds a configuration called initial.
```
primary# ldm add-config initial
```

7. Verify that the configuration is ready to be used.

primary# ldm list-configfactory-default
initial [current]

Configuring a Guest Domain

After the control domain has been configured, create the guest domain to be used as the Oracle Database node. Only one guest domain is created on the physical server that will be the source machine. The other physical server will eventually become the target machine.

The guest domain ldg1 is initially created with the following resources:

Eight GB of memory.
Twenty-four CPUs.
One virtual network interface (vnet0) that is connected to the virtual switch primary-vsw0.
One virtual disk, which appears as c0d0 in the guest domain and is a LUN from the storage array. The domain ldg1 uses LUN 0 of the storage array (c2t6d0).
A second virtual disk, which appears as c0d1 in the guest domain and is a LUN from the storage array. The domain ldg1 uses LUN 1 of the storage array (c2t6d1). This virtual disk is used to store Oracle Database files.

Perform the following steps to create each guest domain:

1. Create the ldg1 guest domain from the control domain on the source machine.

Note: Perform only the ldm set-crypto command if your server is not a SPARC T4-based system.

primary# ldm create ldg1

primary# ldm set-vcpu 24 ldg1
primary# ldm set-memory 8G ldg1
primary# ldm set-crypto 3 ldg1
primary# ldm add-vnet vnet0 primary-vsw0 ldg1
primary# ldm add-vdsdev /dev/dsk/c2t6d0s2  vol1@primary-vds0
primary# ldm add-vdisk vol1 vol1@primary-vds0 ldg1
primary# ldm add-vdsdev /dev/dsk/c2t6d1s2 oradata@primary-vds0
primary# ldm add-vdisk oradata oradata@primary-vds0 ldg1

2. After the domain is created, bind and start the guest domain from the control domain on the source machine by using the following commands.
```
primary# ldm bind ldg1primary# ldm start ldg1
```

3. Determine the console port for the domain.

primary# ldm ls ldg1NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL UPTIME
ldg1             active     -n----  5000    24    8G       0.0%  1h 1m

4. Access the console of the ldg1 domain by using the telnet command.
```
primary# telnet localhost 5000
```

After the guest domain is started, ensure that the appropriate Oracle Solaris OS and patches are installed in the guest domain. You can perform the installation over the network, from a DVD, or by using a DVD ISO image. Refer to the Oracle VM Server for SPARC 2.1 Administration Guide.

Note: LUNs often appear with different names on different servers. LUN names in control domains are not required to be identical. But the control domains on both systems must virtualize the same LUNs (LUN 0 and LUN 1) with the same volume names (vol1 and oradata).

The example in Listing 2 shows how a domain called ldg1 is configured with eight CPUs, 24 GB of memory, and two LUNs from the Sun Storage 2540-M2 storage array.

Listing 2. Example Domain Configuration

primary# ldm ls -l ldg1

NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
ldg1             active     -n----  5000    24    8G       0.0%  1h 1m
SOFTSTATE
Solaris running
MAC
    00:14:4f:fb:96:89
HOSTID
    0x84fb9689
CONTROL
    failure-policy=ignore
DEPENDENCY
    master=
VCPU
    VID    PID    UTIL STRAND
    0      16     0.2%   100%
    1      17     0.0%   100%
    2      18     0.5%   100%
    3      19     0.0%   100%
    4      20     0.0%   100%
    5      21     0.0%   100%
    6      22     0.1%   100%
    7      23     0.0%   100%
    8      24     0.1%   100%
    9      25     0.0%   100%
    10     26     0.0%   100%
    11     27     0.0%   100%
    12     28     0.0%   100%
    13     29     0.0%   100%
    14     30     0.0%   100%
    15     31     0.0%   100%
    16     32     0.0%   100%
    17     33     0.0%   100%
    18     34     0.0%   100%
    19     35     0.0%   100%
    20     36     0.0%   100%
    21     37     0.0%   100%
    22     38     0.0%   100%
    23     39     0.1%   100%
MEMORY
    RA               PA               SIZE
    0x8000000        0x408000000      8G
VARIABLES
    boot-device=vdisk1
    keyboard-layout=US-English
NETWORK
    NAME      SERVICE                ID  DEVICE     MAC                MODE  PVID  VID   MTU   LINKPROP
    vnet1     primary-vsw0@primary   0   network@0  00:14:4f:f9:c0:62         1          1500 
DISK
    NAME      VOLUME                 TOUT ID   DEVICE  SERVER         MPGROUP
    vdisk1    vol1@primary-vds0           0    disk@0  primary
    oradata   oradata@primary-vds0        1    disk@1  primary
VCONS
    NAME      SERVICE                     PORT
    ldg1      primary-vcc0@primary        5000

The example in Listing 3 shows the control domain's configuration. Use the ldm list-services command to verify that the services have been created.

Listing 3. Example Control Domain Configuration

primary# ldm ls-services primary

VCC
    NAME          LDOM     PORT-RANGE
    primary-vcc0  primary  5000-5100
VSW
    NAME          LDOM     MAC               NET-DEV   ID   DEVICE    LINKPROP  DEFAULT-VLAN-ID ...
    primary-vsw0  primary  00:14:4f:f9:32:b0 nxgeg0    0    switch@0  1          1              ...
VDS
    NAME          LDOM     VOLUME   OPTIONS  MPGROUP        DEVICE
    primary-vds0  primary  vol1                             /dev/dsk/c2t6d0s2
                           oradata                          /dev/dsk/c2t6d1s2

Installing Oracle Solaris in the Guest Domain

Ensure that the appropriate release of Oracle Solaris is installed in the guest domain, and verify that the required patches are present.

After Oracle Solaris is installed in the guest domain, the system can be configured and enabled to use the Oracle Database.

Installing Oracle Database

Installation of the Oracle Database software is similar to a standard Oracle Database installation. Start with the installation of Oracle Database and then apply the latest patch set. Use the second LUN to create a UFS or ZFS file system for storing Oracle Database data files.

For more information about using ZFS for an Oracle Database, see Configuring Oracle Solaris ZFS for an Oracle Database.

Preparing the Target Server

Prepare the target server as follows.

Perform the steps described in the following sections to prepare the target machine:
1. Installing the Oracle VM Server for SPARC Software
2. Configuring the Control Domain
Ensure that all virtual I/O services that are used by the domain to be migrated are available on the target server.
Ensure that the target server has access to both LUN 0 and LUN 1 from the same storage.
To add the first LUN (LUN 0) and second LUN (LUN 1) to the target machine, execute the following commands from the control domain of the target server:
```
# ldm add-vdsdev /dev/dsk/c2t6d0s2 vol1@primary-vds0

# ldm add-vdsdev /dev/dsk/c2t6d1s2 oradata@primary-vds0
```
Note: LUNs often appear with different names on different servers. LUN names in control domains are not required to be identical. But the control domains on both systems must virtualize the same LUNs (LUN 0 and LUN 1) with the same volume names (vol1 and oradata).
Ensure that each virtual network device in the domain to be migrated has a corresponding virtual network switch on the target machine:
```
# ldm add-vsw net-dev=nxge0 primary-vsw0 primary
```