How to Make Fibre Channel Storage Available to Oracle Solaris

With the Sun ZFS Storage Appliance

by Andrew Ness, February 2012

How to configure Fibre Channel LUNs in Oracle's Sun ZFS Storage Appliance and integrate them into an Oracle Solaris 10 or Oracle Solaris 11 environment.

Introduction

Oracle Solaris 10 and Oracle Solaris 11 come with a Fibre Channel initiator stack you can configure to integrate Fibre Channel (FC) LUNs presented by the Sun ZFS Storage Appliance into the Oracle Solaris environment. This article describes how to configure the Oracle Solaris Fibre Channel stack and how to configure the Sun ZFS Storage Appliance to set up FC LUNs for access by Oracle Solaris servers. These configurations can be accomplished using the browser user interface (BUI).

The following assumptions have been made in this article:

The root password for the Sun ZFS Storage Appliance is known.
The IP address or host name of the Sun ZFS Storage Appliance is known.
The network used by the Sun ZFS Storage Appliance has been configured.
The Sun ZFS Storage Appliance has pools configured with sufficient free space available.
The root password for the Oracle Solaris server is known.
The Sun ZFS Storage Appliance is already connected to a Fibre Channel switch.
The appropriate zoning has been configured on the FC switch to allow the Oracle Solaris host to access the Sun ZFS Storage Appliance.

Configuring the Oracle Solaris FC Stack

To identify the Sun ZFS Storage Appliance and the Oracle Solaris server to each other, the FC World Wide Number (WWN) for each must be registered with the other. You must determine the WWNs for some forms of FC zoning implemented on the FC switch.

The FC WWN of the host is used to identify the host to the Sun ZFS Storage Appliance and is needed to complete the configuration procedure in this article.

The WWNs are taken from the FC Host Bus Adapters (HBAs) installed in the Oracle Solaris host and the Sun ZFS Storage Appliance.

In order to configure the Oracle Solaris FC stack, you will need to know the WWNs of the Sun ZFS Storage Appliance. In a traditional dual-fabric Storage Area Network (SAN), the Sun ZFS Storage Appliance will have at least one FC port attached to each fabric. Thus, you must determine at least two FC WWNs.

Identifying the Sun ZFS Storage Appliance FC WWNs

First, you need to establish a management session with the Sun ZFS Storage Appliance.

1. Enter an address in the address field of a Web browser that includes the IP address or host name of the Sun ZFS Storage Appliance as shown in the following URL:
https://<ip-address or host name>:215
The login dialog box is displayed.

Figure 1. Sun ZFS Storage Appliance login
2. Enter a username and password and click LOGIN.

Once you have successfully logged in to the BUI, you can identify the WWNs through the Configuration tab.
3. Click on Configuration > SAN > Fibre Channel Ports.

Figure 2. Fibre Channel port configuration for SAN

The FC ports installed in the Sun ZFS Storage Appliance are shown. Since there is only one discovered port per HBA channel, this must be the HBA channel itself.

In the preceding case, port 1 has the WWN 21:00:00:e0:8b:92:a1:cf and port 2 has the WWN 21:01:00:e0:8b:b2:a1:cf.

The FC channel ports should be set to Target in the list box to the right of each FC port box. If this is not the case, the FC ports may be in use for another purpose. Do not change the setting until you have investigated the reason. (One possible reason may be for NDMP backups.)

Identifying the Oracle Solaris Host HBA WWNs

If the Oracle Solaris host is already attached to the FC switch by the appropriate cables, use the following command to identify the WWNs.

1. To find the WWNs for the host, enter:


root@solaris:~# cfgadm -al -o show_FCP_dev
Ap_Id                  Type         Receptacle   Occupant     Condition
[...]
c8                     fc-fabric    connected    unconfigured unknown
c9                     fc-fabric    connected    unconfigured unknown
root@solaris:~#

From this output, the controller numbers you need are c8 and c9. You can also see that both ports are attached to an FC switch, as the port type is fc-fabric. Next, interrogate these controllers to determine what WWNs are discovered.

2. If the HBA ports are not being used to access any other FC-connected devices, the following command can be used to determine the WWNs.


root@solaris:~# prtconf -vp | grep port-wwn
      port-wwn:  210000e0.8b89bf8e
      port-wwn:  210100e0.8ba9bf8e
root@solaris:~#

Alternatively, if FC devices are being accessed, the following command will show the FC HBA WWNs.


root@solaris:~# luxadm -e dump_map /dev/cfg/c8
Pos  Port_ID Hard_Ad Port WWN         Node WWN           Type
0    2000    0       xxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxx 0x0  (Disk 
     device)
[...]
3    25600   0       2100000e08b89bf8e 200000e08b89bf8e 0x1f (Unknown    
     type, Host Bus Adapter)
root@solaris:~#

The last entry shown as type 0x1f (Unknown type, Host Bus Adapter) gives the appropriate WWN under the port WWN entry. Repeat the command substituting /dev/cfg/c8 for the other controllers identified in step 1.

3. From the output, you can see that c8 has the WWN 21:00:00:00:e0:8b:89:bf:8e and c9 has WWN 21:01:00:e0:8b:a9:bf:8e.

The Sun ZFS Storage Appliance HBA and the Oracle Solaris host HBA WWNs can then be used to configure any FC switch zoning.

Once this has been completed, you can verify the proper zoning by running the following command:
```
root@solaris:~# cfgadm -al -o show_FCP_dev c8 c9
Ap_Id                  Type         Receptacle   Occupant     Condition
[...]
c8                     fc-fabric    connected    configured   unknown
c8::210100e08bb2a1cf   unknown      connected    unconfigured unknown
c9                     fc-fabric    connected    configured   unknown
c9::210000e08b92a1cf   unknown      connected    unconfigured unknown
root@solaris:~# 
```
You can now see the WWNs presented by the Sun ZFS Storage Appliance that are accessible by the Oracle Solaris host.

Configuring the Sun ZFS Storage Appliance Using the Browser User Interface

As a unified storage platform, the Sun ZFS Storage Appliance supports access to block-protocol LUNs using iSCSI and Fibre Channel protocols. This section describes how to use the Sun ZFS Storage Appliance BUI to configure the Sun ZFS Storage Appliance to recognize the Oracle Solaris host and present FC LUNs to it.

Defining an FC Target Group

A target group is created on the Sun ZFS Storage Appliance to define the ports and the protocol by which the LUN to be presented to the Oracle Solaris server is accessed. For this example, an FC target group is created.

To define an FC target group on the Sun ZFS Storage Appliance, complete these steps:

1. Click Configuration > SAN to display the Storage Area Network (SAN) screen, as shown in Figure 3.
2. Click the Targets tab at the right and then select Fibre Channel Ports at the top of the left panel, as shown in Figure 3.
3. Place the mouse in the Fibre Channel Ports box and a Move icon () will appear in the far left, as shown in Figure 3.

Figure 3. Selecting the Fibre Channel port
4. Click on the Move icon and drag the box to the Fibre Channel Target Groups box, as demonstrated in Figure 4.

Figure 4. Dragging the Fibre Channel port entry
5. Drop the entry in the orange box to create a new target group. The group is created and named automatically as targets-n where n is an integer. See the example targets-0 in Figure 5.

Figure 5. Newly created Fibre Channel target group
6. Move the cursor over the entry for the new target group. Two icons appear to the right of the Fibre Channel Target Groups box, as shown in Figure 6.

Figure 6. Selecting the FC target group
7. To rename the new target group targets-0, click the Edit icon () to display the dialog box shown in Figure 7.

Figure 7. Renaming the FC Target Group
8. In the Name field, replace the default name with your preferred name for the new FC target group and click OK. For this example, the name FC-PortGroup replaces targets-0. In this window, you can also add the second FC target port by clicking in the box to the left of the WWN you are choosing. In the example in Figure 7, this second port is identified as PCIe 1: Port 2.
9. Click OK to save the changes.
10. Click APPLY. The changes are shown in the Fibre Channel Target Groups panel in Figure 8.

Figure 8. New FC target group with changes applied

Defining an FC Initiator

An FC initiator is defined to allow access from one or more servers to particular volumes. Permissions to volumes should be configured so that a minimum number of FC initiators are allowed to access a particular volume. If more than one host can write to a given volume concurrently and a non-shared file system is used, inconsistencies may occur in file system caches on the hosts that can ultimately lead to corruption of the on-disk image. Typically a single initiator is given access to a volume, unless a specialized cluster file system is being used.

The FC initiator serves to define the "host" from the point of view of the Sun ZFS Storage Appliance. In a traditional dual-fabric SAN, the host will be defined by at least two FC initiators. The FC initiator definition contains the host WWNs. To identify the Oracle Solaris server to the Sun ZFS Storage Appliance, the Oracle Solaris FC initiator WWNs must then be registered with the appliance by completing the following steps.

1. Click Configuration > SAN to display the Storage Area Network (SAN) screen shown in Figure 9.
2. Click the Initiators tab at the right and then select Fibre Channel Initiators at the top of the left panel, as shown in Figure 9.

Figure 9. Selecting the SAN configuration
3. Click the icon to the left of Fibre Channel Initiators to display the New Fibre Channel Initiator dialog box shown in Figure 10.

If the zoning has been configured on the FC switches, the WWNs of the Oracle Solaris host should be displayed (assuming they are not assigned to an alias already).
4. Click on one of the WWNs (if they are displayed) at the bottom of the dialog box to prepopulate the World Wide Name or type the appropriate WWN in to the World Wide Name box.
5. Enter a more meaningful symbolic name in the Alias box.

Figure 10. New Fibre Channel initiator definition
6. Click OK.
7. Repeat the preceding steps for the other WWN(s) that refer to the Oracle Solaris host.

Defining the FC Initiator Group

Related FC initiators are combined into logical groups to allow single commands to execute on multiple FC initiators, for example, assigning LUN access to all FC initiators in a group with one command. For the following example, the FC initiator group will contain two initiators. Note that in a cluster, where multiple servers are treated as a single logical entity, the initiator group may contain many more initiators.

To create an FC initiator group, complete these steps:

1. Select Configuration > SAN to display the Storage Area Network (SAN) screen.
2. Select the Initiators tab at the right and then click Fibre Channel Initiators at the top of the left panel.
3. Place the cursor over the entry for one of the FC initiators created in the previous section. The Move icon () appears to the left of the entry, as shown in Figure 11.

Figure 11. Displaying the Move icon for the new FC initiator
4. Click the Move icon and drag it to the Fibre Channel Initiator Groups panel on the right. A new entry appears (highlighted in yellow) at the bottom of the Fibre Channel Initiators Groups panel shown in Figure 12.

Figure 12. Creating the FC initiator group
5. Move the cursor over the new entry box and release the mouse button. A new FC initiator group is created with the name initiators-n, where n is an integer, as shown in Figure 13.
6. Move the cursor over the entry for the new initiator group. Several icons appear to the right of the target initiator group box, as shown in Figure 13.

Figure 13. Selecting the Fibre Channel initiator group
7. Click the Edit icon () to display the dialog box shown in Figure 14.
8. In the Name field, replace the new initiator group's default name with a chosen name and click OK. For this example, the name sol-server is used.

In this dialog box, you can add the other FC initiator(s) to the group by clicking the checkbox to the left of the WWN.

Figure 14. Renaming and completing the FC initiator group
9. Click APPLY on the SAN configuration screen to confirm all the modifications, as shown in Figure 15.

Figure 15. Completed FC initiator configuration

Defining a Sun ZFS Storage Appliance Project

To group related volumes, you can define a project in the Sun ZFS Storage Appliance. A project allows inheritance of properties for file systems and LUNs presented from the project. A project also allows you to apply quotas and reservations.

To create a project, complete the steps below:

1. Select Shares > Projects to display the Projects screen, as shown in Figure 16.

Figure 16. Viewing a project
2. Click the icon to the left of Projects at the top of the left panel to display the Create Project dialog box shown in Figure 17.

Figure 17. Create Project dialog box
3. To create a new project, enter a name for the project and click APPLY. A new project appears in the Projects list in the left panel.
4. Select the new project to view the components that comprise the project, as shown in Figure 18.

Figure 18. Displaying the new project sol-project

Defining a Sun ZFS Storage Appliance LUN

Next, you will create a LUN from an existing pool of storage resources that the Oracle Solaris server will access. In the following example, a thin-provisioned 64-GB LUN called DocArchive1 is created.

The target group will be the FC target group, FC-PortGroup, which was created in the section Defining an FC Target Group, to ensure that this LUN can be accessed using the FC protocol. The initiator group defined in the section Defining the FC Initiator Group, sol-server, will be used to ensure that only the server(s) defined in the sol-server group can access this LUN. (In this example, this group contains only one server.)

To create a target group, complete the following steps:

1. Select Shares > Projects to display the Projects screen.
2. In the Projects panel at the left, select the project. Then select LUNs at the top of the panel at the right, as shown in Figure 19.

Figure 19. Selecting Shares > Project > LUNs
3. Click the icon to the left of LUNs to display the Create LUN dialog box shown in Figure 20.
4. Enter values as appropriate to set up the LUN. For this example, set Name to DocArchive1, Volume size to 64 G, and select the Thin provisioned checkbox. Set Target Group to the FC target group FC-PortGroup and set Initiator Group to sol-server. Set Volume block size to 32k because the volume will hold an Oracle Solaris ZFS file system.

Figure 20. New LUN dialog box
5. Click APPLY to create the LUN and make it available to the Oracle Solaris server.

Configuring the LUN for Use by the Oracle Solaris Server

Now that the LUN is prepared and available to the FC initiator group, the LUN must be configured for use by the Oracle Solaris server by completing the following steps:

1. Initiate an Oracle Solaris FC session with the Sun ZFS Storage Appliance, as shown in Listing 1. Since the LUN was created prior to initiating the FC session, it will be automatically enabled.

Listing 1. Initiating an Oracle Solaris FC session



 root@solaris:~# cfgadm -al c8 c9
root@solaris:~# cfgadm -c configure c8::210100e08bb2a1cf
root@solaris:~# cfgadm -c configure c9::210000e08b92a1cf
root@solaris:~# cfgadm -al -o show_FCP_dev c8 c9
Ap_Id                  Type         Receptacle   Occupant     Condition
c8                     fc-fabric    connected    configured   unknown
c8::210100e08bb2a1cf,0 disk         connected    configured   unknown
c9                     fc-fabric    connected    configured   unknown
c9::210000e08b92a1cf,0 disk         connected    configured   unknown
root@solaris:~#

2. Verify access to the FC LUN, as shown in Listing 2.

Listing 2. Verifying access to the FC LUN





root@solaris:~# devfsadm -c ssd
root@solaris:~# tail /var/adm/messages
[...]
Nov 30 06:30:00 solaris last message repeated 2 times
Nov 30 06:32:22 solaris scsi: [ID 799468 kern.info] ssd42 at 
scsi_vhci0: name g600144f0f05e906c00004ed6096d0001, bus address 
g600144f0f05e906c00004ed6096d0001
Nov 30 06:32:22 solaris genunix: [ID 936769 kern.info] ssd42 is 
/scsi_vhci/ssd@g600144f0f05e906c00004ed6096d0001
Nov 30 06:32:22 solaris scsi: [ID 107833 kern.warning] WARNING: 
/scsi_vhci/ssd@g600144f0f05e906c00004ed6096d0001 (ssd42):
Nov 30 06:32:22 solaris     Corrupt label; wrong magic number
Nov 30 06:32:22 solaris genunix: [ID 408114 kern.info] 
/scsi_vhci/ssd@g600144f0f05e906c00004ed6096d0001 (ssd42) online
Nov 30 06:32:22 solaris genunix: [ID 834635 kern.info] 
/scsi_vhci/ssd@g600144f0f05e906c00004ed6096d0001 (ssd42) 
multipath status: degraded, path 
/pci@1e,600000/SUNW,qlc@2/fp@0,0 (fp1) to target address: 
w210100e08bb2a1cf,0 is online Load balancing: round-robin
Nov 30 06:32:22 solaris scsi: [ID 107833 kern.warning] WARNING: 
/scsi_vhci/ssd@g600144f0f05e906c00004ed6096d0001 (ssd42):
Nov 30 06:32:22 solaris     Corrupt label; wrong magic number
Nov 30 06:32:49 solaris genunix: [ID 834635 kern.info] 
/scsi_vhci/ssd@g600144f0f05e906c00004ed6096d0001 (ssd42) 
multipath status: optimal, path 
/pci@1e,600000/SUNW,qlc@2,1/fp@0,0 (fp3) to target address: 
w210000e08b92a1cf,0 is online Load balancing: round-robin
[...]

In this example, the multipath status is shown as degraded initially because only one path had been identified at that point. Further on, the multipath status changes to optimal due to the presence of multiple paths to the volume.

The disk device is now similarly available to an internal server disk.

3. Format the LUN as shown in Listing 3.

Listing 3. Formatting the LUN


   

root@solaris:~# zfs createzpool create docarchive1 \
c1t600144F0F05E906C00004ED6096D0001d0
root@solaris:~# zfs list
NAME                    USED  AVAIL  REFER  MOUNTPOINT
docarchive1              72K  62.5G    21K  /docarchive1
rpool                  25.4G  41.5G    97K  /rpool
rpool/ROOT             22.3G  41.5G    21K  legacy
[...]
root@solaris:~# zfs create docarchive1/index
root@solaris:~# zfs create docarchive1/data
root@solaris:~# zfs create docarchive1/logs
root@solaris:~# zfs list
NAME                    USED  AVAIL  REFER  MOUNTPOINT
docarchive1             159K  62.5G    24K  /docarchive1
docarchive1/data         21K  62.5G    21K  /docarchive1/data
docarchive1/index        21K  62.5G    21K  /docarchive1/index
docarchive1/logs         21K  62.5G    21K  /docarchive1/logs
rpool                  25.4G  41.5G    97K  /rpool
rpool/ROOT             22.3G  41.5G    21K  legacy
[...]

4. Build an Oracle Solaris ZFS file system on the prepared LUN by creating a new ZFS pool, adding this device to the ZFS pool, and creating the ZFS file system, as shown in the example in Listing 4.

Listing 4. Building an Oracle Solaris ZFS file system




root@solaris:~# zfs createzpool create docarchive1 \
c1t600144F0F05E906C00004ED6096D0001d0
root@solaris:~# zfs list
NAME                    USED  AVAIL  REFER  MOUNTPOINT
docarchive1              72K  62.5G    21K  /docarchive1
rpool                  25.4G  41.5G    97K  /rpool
rpool/ROOT             22.3G  41.5G    21K  legacy
[...]
root@solaris:~# zfs create docarchive1/index
root@solaris:~# zfs create docarchive1/data
root@solaris:~# zfs create docarchive1/logs
root@solaris:~# zfs list
NAME                    USED  AVAIL  REFER  MOUNTPOINT
docarchive1             159K  62.5G    24K  /docarchive1
docarchive1/data         21K  62.5G    21K  /docarchive1/data
docarchive1/index        21K  62.5G    21K  /docarchive1/index
docarchive1/logs         21K  62.5G    21K  /docarchive1/logs
rpool                  25.4G  41.5G    97K  /rpool
rpool/ROOT             22.3G  41.5G    21K  legacy
[...]

The final two lines of the output from the df(1) command show that approximately 64 GB of new space is now available.

Conclusion

This article has described how to configure the FC initiator stack supplied with Oracle Solaris 10 and Oracle Solaris 11 to enable an Oracle Solaris server to access Fibre Channel LUNs presented by the Sun ZFS Storage Appliance.

For More Information

Here are resources for Oracle's Sun ZFS Storage Appliance and Oracle Solaris ZFS:

Sun ZFS Storage Appliance Administration Guide: Oracle Unified Storage Systems documentation libraries or http://download.oracle.com/docs/cd/E22471_01/index.html

Here are some additional resources for Oracle Solaris:

Download Oracle Solaris 11
Access Oracle Solaris 11 product documentation
Access all Oracle Solaris 11 how-to guides
Learn more with Oracle Solaris 11 training and support
See the official Oracle Solaris blog
Check out The Observatory Blog for Oracle Solaris tips and tricks

Revision 1.0, 02/24/2012