How to Script Oracle Solaris 11 Zones Creation for a Network-in-a-Box Configuration

Through this "network-in-a-box" example, learn how to efficiently create Oracle Solaris Zones by cloning them and then use scripts to set up a virtual network.

Published October 2012 (Updated September 2014, updated February 2020)

This article describes a new capability introduced in Oracle Solaris 11.2 that simplifies using scripting to create Oracle Solaris Zones that have multiple network interfaces.

Let's begin with some background on

• The methods for creating zones
• The types of Oracle Solaris Zones network configurations
• The methods for scripting a network interface configuration

Methods for Creating Zones

There are two basic ways to create a zone. You can create it from scratch or you can create it from a clone. Creating from a clone is faster.

If you create a zone from scratch, you will need to transfer packages from the repository (or if a package has already been downloaded, you can get it locally). And then on first boot, approximately 130+ services will need to be configured.

When a zone is cloned, the package installation has already been completed, and most of the services have already been configured. Cloning requires some configuration, but only a small subset of the services is impacted.

If you are creating a dynamic environment where you create zones on demand, cloning is a good approach.

Types of Oracle Solaris Zones Network Configurations

There are two types of networking configurations for zones: shared-IP stack and exclusive-IP stack. The shared-IP stack was developed as a time-to-market facilitation when zones were first introduced in Oracle Solaris 10. However with Oracle Solaris 11 virtual NICs (VNICs), the default becomes exclusive-IP stack, because it allows Oracle Solaris 11 VNICs to much more closely mirror physical NICs.

Keeping the architectural model the same for VNICs as for physical NICs means that application developers don't have to think about any new set of issues. And applications don't have to worry about whether they are on a physical NIC or a VNIC—things work the same on both, the security model is the same, and the commands used to operate on the interfaces are the same (although there are some additional commands, due to additional capabilities of VNICs).

For this article, we deal only with exclusive-IP stack interfaces for zones.

Methods for Scripting a Network Interface Configuration

There are a variety of ways to script the configuration of a network interface for a zone. If you have only one network interface, these are your four choices:

1. You can configure a single network interface for Oracle Solaris 11 Zones using the new Oracle Solaris 11 approach for specifying all installation-time system configuration settings. You use the sysconfig create-profile command to create a file to configure a system.

   This command creates an XML file, so if you want to script creating a zone with one interface, you could create a "template" configuration file (typically called sc_profile.xml) by answering all the questions as you go through the setup. You could then edit that file and for those values that you want to be able to adjust on the fly, you could replace the values that you entered with variable names. For the simplest case, that might require replacing the node name, the IP address and mask, the interface to assign that IP address to, and maybe the default router. That is an easy to accomplish with scripting, and we'll see how below.

2. Oracle Solaris 10 actually had a very clean way to specify all the configuration information in a simple text file called sysidcfg, using one attribute per line.

   You can start with a sysidcfg file from an existing system, for example, to create a template. Then, in the template, use variables to replace any values that you want to be able to adjust. Then create a script to read in values, which will be populated in the sysidcfg template you created. Take the resulting file and use the js2ai(1M) utility to convert the sysidcfg file to an sc_profile.xml configuration profile. The latter can then be used by Oracle Solaris 11 zone commands to apply the final configuration to your zone. The only drawback with this approach is that js2ai can only handle a sysidcfg file with one network interface.

3. Another approach is to create a zone and then log in to the new zone and use ipadm to configure the interface. This isn't the best solution for scripting zone creation, because you would like to do everything from the global zone, rather than having to run additional scripts in the new zone you created.
4. Finally, you can use the allowed-address property, which can be set through zonecfg when setting up the zone from the global zone. The allowed-address property does more than just assign an IP address; it also ensures that only packets with a source address that matches the value of allowed-address will be allowed through the interface. So this approach would impact routing through the zone.

For setting up multiple interfaces, you can use approach 1, 2, or 4. Approaches 2 and 4 are both a bit awkward for various reasons. Approach 1, using sc_profile.xml, uses one mechanism to configure all the settings that initially set up an Oracle Solaris environment, so we are going to use that approach in this article. By the way, support for configuring multiple interfaces through the sc_profile.xml file is new with Oracle Solaris 11.2.

Example of Creating a Network in a Box

Oracle Solaris 11 provides new networking features for network virtualization. Network virtualization takes server virtualization to the next level by providing the ability to virtualize entire network topologies of servers, routers, switches, and firewalls all running on a single platform and requiring no additional investment in networking hardware. Network virtualization can be used for a variety of purposes, from prototyping, to developing and testing, to service deployment.

For this example, we are going to look at a complex example where we have multiple different virtual subnets in a system. If you need to create a number of zones with network interfaces all on the same subnet, and you are associating all those VNICs to the same physical NIC, Oracle Solaris will automatically create a switch for you so that traffic can flow between those zones. However, for a more-complex environment—say, one in which all the zones are on the same subnet, but the VNICs are associated with two different physical NICs—then the situation gets a bit trickier, and you will need to think in terms of etherstubs and zone routers, as described below, below to allow networking traffic between zones to stay within the system. The example illustrated in this article provides the background you need to tackle more-elaborate configurations and implement them through scripting.

Let's first consider a logical depiction of a specific set of services we want to offer on a single system.

Figure 1 shows an example of a multitiered application. In this example, there are two web servers running on dedicated systems that are exposed to the external network, and there are several database servers running on dedicated systems that are not directly connected to the external network.

Figure 1. Network to Be Virtualized on a Single Server

A good place to start the process of configuring this physical scenario on a single Oracle Solaris 11 system would be to name all the entities that we must configure. So we will redraw the diagram the way it will be implemented on Oracle Solaris 11.2, as shown in Figure 2, with an emphasis on all the component and interface names and IP addresses.

Figure 2. Converting from a Physical Topology to a Virtual Network-in-a-Box Topology

In Figure 2, the web servers share a back-end etherstub, webswitch1. Technically, an etherstub is not a virtual switch, but it enables a switch to be created. So labeling an etherstub as a switch makes the diagram clear without adding extra detail about the underlying implementation.

The database servers also share a back-end etherstub labeled dbswitch1. The network will be configured so the database servers running in dedicated Oracle Solaris Zones (zdb1, zdb2, and zdb3) have no direct access to the physical network, which can be the basis for insulating them from direct outside communications.

The single physical interface is represented in this example as net0.

Overview of the Solution

Before we get to the scripting, we need to prep the system as follows:

• Create the Oracle Solaris etherstubs (Step 1)
• Create the ZFS data set for storing the Oracle Solaris Zones (Step 2)
• Create an Oracle Solaris Zone for cloning (Step 3)
• Create the configuration profiles (Step 4)

Then we will walk through configuring a zone manually to see how that is done (Step 5).

We are then going to use some of the scripts provided in the appendix to create part of the complex topology shown in Figure 2 by doing the following (Step 6):

• Create zdb1, one of the database zones, with an interface connected to dbswitch1.
• Create the zone that has two network interfaces to two switches: zrouter.
• Create the web server zone zweb1, which has two network interfaces: one to a VNIC on a physical NIC to the outside world and the other to a switch, webswitch1.

Finally, we will perform some necessary post-processing (Step 7) and then explore the network-in-a-box configuration we created (Step 8).

Step 1: Create the Etherstubs

In this step, we create the two etherstubs that are used to enable communication between the database zones and between the web server zones shown in Figure 2. Etherstubs, as mentioned above, are a mechanism that facilitates creating switches within virtual networks in an Oracle Solaris environment.

Let's start by looking at the physical network on this system:

# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         100    full      nge0
net1              Ethernet             unknown    0      unknown   nge1

Listing 1

As we see in Listing 1, there are two physical interfaces, nge0 and nge1. No cable is plugged into nge1. This will be the last time we reference these interfaces by their underlying driver names. With Oracle Solaris 11, interfaces have a parallel and simple naming scheme: net0 for the first, net1 for the second, and so forth. Think of those as names that are consistent across all Oracle Solaris 11 environments. This is a useful simplification, because we no longer have to worry about the actual physical NIC hardware link names for various hardware platforms, such as nge0, bge0, e1000g, and so on. When we want to configure anything to do with the first physical interface, we just use net0.

We start by creating two etherstubs that will enable the virtual switching that occurs to match our diagrammed architecture:

# dladm create-etherstub webswitch1
# dladm create-etherstub dbswitch1 

Now check what we have created:

# dladm show-etherstub -Z 
LINK                ZONE 
dbswitch1           global 
webswitch1          global 

In Oracle Solaris 11, each Oracle Solaris Zone comes with a VNIC, net0, already enabled. For the purposes of this example, we name the VNICs in Figure 2 to reflect their place in the overall architecture. So rather than label all the interfaces in the diagram net0, net1, and so on, we use more descriptive names, such as vdb1, vweb1, and so on.

This overloading of the identifier net0 can be confusing—net0 is the underlying physical interface for the first interface found by the system, and when we create a zone, we automatically get a VNIC called net0. Those are two different naming constructs.

Step 2: Create a ZFS Data Set for Storing the Oracle Solaris Zones

We are going to create a single Oracle Solaris Zone and then, to reduce the time to create additional ones, we will clone the first zone to make the remaining zones needed for our example.

Cloned zones require sharing a ZFS data set, so creating the data set is the first step:

# zfs create -o mountpoint=/zonefs rpool/zonefs 
# chmod 700 /zonefs

Step 3: Create an Oracle Solaris Zone for Cloning

As mentioned earlier, it is much faster to clone an Oracle Solaris Zone than to create one from scratch, because building an image from packages takes longer than, in essence, copying an existing zone. We will use the cloning technique in this example to first create one Oracle Solaris Zone and then clone it.

First, configure the Oracle Solaris Zone that we will clone, zclone, as shown in Listing 2:

# zonecfg -z zclone 
zclone: No such zone configured 
Use 'create' to begin configuring a new zone. 
zonecfg:zclone> create 
create: Using system default template 'SYSdefault' 
zonecfg:zclone> set zonepath=/zonefs/zclone 
zonecfg:zclone> set ip-type=exclusive 
zonecfg:zclone> exit 

Listing 2

The default setting for ip-type for Oracle Solaris 11 is exclusive so we did not actually have to specify that in Listing 2.

Now install zclone:

# zoneadm -z zclone install 
A ZFS file system has been created for this zone. 
Progress being logged to /var/log/zones/zoneadm.20111007T223740Z.zclone.install 
. 
. 
. 
Done: Installation completed in 407.653 seconds.
    Next Steps: Boot the zone, then log into the zone console
               (zlogin -C) to complete the configuration process. 

Note: Installation times vary depending on the network and the Oracle Solaris Image Packaging System (IPS) repository.

Now boot zclone, as shown in Listing 3.

# zoneadm -z zclone boot; zlogin -C zclone 
[Connected to zone 'zclone' console]
 Loading smf(5) service descriptions: 134/134 

Listing 3

In Listing 3, we put the zlogin command on the same line as the boot command so that the zlogin command will be immediately executed, and you will see the zclone zone boot.

The output should eventually disappear and be replaced by the initial screen of the System Configuration Tool (see Figure 3), where you can do the final configuration.

Important: Do not use the tool. At this point, we won't do further configuration, so escape to the global zone using ~. (press the tilde key followed by the period key).

Figure 3. System Configuration Tool

Then halt zclone:

# zoneadm -z zclone halt

To review, we created an Oracle Solaris Zone, zclone, which has gone through the initial installation and first boot, which is where most of the Oracle Solaris Service Management Facility services were configured. At that point, we stopped the process, because we don't want to further configure zclone.

Step 4: Create the Configuration Profiles

We stopped before the configuration step, so now we will create a file that contains configuration settings that can be applied to our cloned zones to create the zones we need for this example.

We will start by running sysconfig with the option create-profile—which will bring up the same System Configuration Tool screen that we just escaped from—to allow us to save all the settings in a file that we can customize.

Run the following command, where <NetInABoxDir> is the directory where the sc_profile.xml file will be stored:

# sysconfig create-profile -o <NetInABoxDir>

Do the following and fill out all the screens until the command completes:

• Use zweb1 as the system name.
• Select manual networking.
• Don't specify a default route.
• Don't select DNS (we will discuss why below).
• Create a user called demo.

Then make a copy of the file:

# cp NetInABox/sc_profile.xml OneInterfaceTemplate.xml

Now we have a configuration profile, OneInterfaceTemplate.xml, based on the values we specified. We want to replace some of those values with variables. Then, later you could run the file through a script (Listing 17 in the appendix) and replace things such as the name of the zone, the IP address of the zone, and so on with values to match Figure 2.

Bring up a text editor and open OneInterfaceTemplate.xml. Find the section where the nodename is defined (see Listing 4).

<service version="1" type="service" name="system/identity">
    <instance enabled="true" name="node">
      <property_group type="application" name="config">
        <propval type="astring" name="nodename" value="zweb1"/>
      </property_group>
    </instance>
</service>

Listing 4

Note: Make sure you are using a text editor that doesn't replace double quotes with Unicode left double quote and right double quote. You just want pure ASCII text.

We want to turn the system name into a variable, so replace zweb1 with ZONENAME as follows:

<propval type="astring" name="nodename" value="ZONENAME"/>

Now scroll down to the IPv4 section where the IP address and mask are found (see Listing 5):

<property_group type="application" name="install_ipv4_interface">
   <propval type="net_address_v4" name="static_address" value="129.200.9.2/24"/>
   <propval type="astring" name="name" value="net0/v4"/>
   <propval type="astring" name="address_type" value="static"/>
</property_group>    

Listing 5

We are going to make five changes to the XML block shown in Listing 5:

1. Replace the type (application) with ipv4_interface.
2. Replace the name of the property group (install_ipv4_interface) with install_ipv4_interface_0.

   These first two changes are the core of the Oracle Solaris 11.2 functionality that allows you to specify multiple interfaces in a profile. We specify only interface_0 here and later will add interface_1, but you could continue to increment that number to add as many interfaces as needed.

3. Replace the IP address and mask (129.200.9.2) with IPADDRandMASK_1.
4. Replace the NIC name (net0) with VNIC1.
5. Add a line to set the default route through the DEFAULTROUTE variable.

You should now have the block shown in Listing 6:

<property_group type="ipv4_interface" name="install_ipv4_interface_0">
   <propval type="net_address_v4" name="static_address" value="IPADDRandMASK_1"/>
   <propval type="astring" name="name" value="VNIC1/v4"/>
   <propval type="astring" name="address_type" value="static"/>
   <propval type="net_address_v4" name="default_route" value="DEFAULTROUTE"/>
</property_group>

Listing 6

Save the file OneInterfaceTemplate.xml. We now have a template for configuring a zone with one network interface.

Make a copy of the file:

# cp OneInterfaceTemplate.xml TwoInterfaceTemplate.xml

Now we are going to make a template for use when we want to create a zone with two network interfaces.

Edit TwoInterfaceTemplate.xml and then copy the IP address property group block and add the copied block after the original block, as shown in Listing 7:

<property_group type="ipv4_interface" name="install_ipv4_interface_0">
   <propval type="net_address_v4" name="static_address" value="IPADDRandMASK_1"/>
   <propval type="astring" name="name" value="VNIC1/v4"/>
   <propval type="astring" name="address_type" value="static"/>
   <propval type="net_address_v4" name="default_route" value="DEFAULTROUTE"/>
</property_group>
<property_group type="ipv4_interface" name="install_ipv4_interface_0">
   <propval type="net_address_v4" name="static_address" value="IPADDRandMASK_1"/>
   <propval type="astring" name="name" value="VNIC1/v4"/>
   <propval type="astring" name="address_type" value="static"/>
   <propval type="net_address_v4" name="default_route" value="DEFAULTROUTE"/>
</property_group>

Listing 7

Make the four edits described below to the second property group shown in Listing 7:

1. Replace the type name (install_ipv4_interface_0) with install_ipv4_interface_1.
2. Replace the IP address and mask (IPADDRandMASK_1) with IPADDRandMASK_2.
3. Replace the variable VNIC1 with VNIC2.
4. Remove the line where default_route is set.

The result should be what is shown in Listing 8:

<property_group type="ipv4_interface" name="install_ipv4_interface_0">
   <propval type="net_address_v4" name="static_address" value="IPADDRandMASK_1"/>
   <propval type="astring" name="name" value="VNIC1/v4"/>
   <propval type="astring" name="address_type" value="static"/>
   <propval type="net_address_v4" name="default_route" value="DEFAULTROUTE"/>
</property_group>
<property_group type="ipv4_interface" name="install_ipv4_interface_1">
   <propval type="net_address_v4" name="static_address" value="IPADDRandMASK_2"/>
   <propval type="astring" name="name" value="VNIC2/v4"/>
   <propval type="astring" name="address_type" value="static"/>
</property_group>

Listing 8

Save TwoInterfaceTemplate.xml and exit the editor.

We have defined six new variables: ZONENAME, IPADDRandMASK_1, IPADDRandMASK_2, VNIC1, VNIC2, and DEFAULTROUTE. There is a very simple script in the appendix (Listing 17) that takes as input the values to assign to these variables plus the name of the profile XML template, and then it outputs the result to another XML file that you would use to create specific zones.

However, for experimenting with this network-in-a-box environment, all you need is the three zones pictured in Figure 4. Rather that work out how to pass eight arguments to one version of the script and six to another, in the correct order, it is probably easier and less error-prone to just manually do the replacement for the three zones. And if you were going to use these scripts in a real production environment, you would probably combine the functionality of setting up the configuration profile through the same script that then creates the zone.

The scaled down environment we want to build is just the top three zones: zweb1, zrouter, and zdb1.

Figure 4. What We Will Implement

To configure zweb1, first make a copy of TwoInterfaceTemplate.xml.

# cp TwoInterfaceTemplate.xml zweb1profile.xml

Edit zweb1profile.xml and make the value substitutions shown in Table 1. Your choice for IPADDRandMASK_1 should be a real address known within your organization. Save the resulting customized profile as zweb1profile.xml.

Notes:

• Don't forget the mask (for example, 129.200.9.2/23) for the IP address attached on the external network. In our particular case, our network mask is 254, not the more typical 255, so we select 23 to correspond to the 23 bits set.
• When specifying the DEFAULTROUTE, don't specify a mask. Also, routers typically have an address such as nnn.nnn.nnn.1 or nnn.nnn.nnn.254. In our case, the router for 129.200.9 is at 129.200.9.1.

To configure zrouter, again start by making a copy of TwoInterfaceTemplate.xml.

# cp TwoInterfaceTemplate.xml zrouterprofile.xml

Edit zrouterprofile.xml to match the settings shown in Table 2. You don't need the DEFAULTROUTE, so remove that line. For the internal networks (192.168.0 and 192.168.7), we will use a conventional 24-bit mask.

Finally, to configure zdb1, make another copy of OneInterfaceTemplate.xml:

# cp OneInterfaceTemplate.xml zdb1profile.xml

And then edit zdb1profile.xml to match the settings shown in Table 3.

Step 5: Create the zweb1 Zone Manually

We will now walk through configuring zone zweb1 manually.

You created the profile zweb1profile in the Step 4, with the fields filled in as shown in Table 1. Now create the zweb1 zone from the clone, zclone.

# zonecfg -z zweb1 create -t zclone

Associate some storage with the cloned zone:

# zonecfg -z zweb1 set zonepath=/zonefs/zweb1

Now start the zone configuration process. The command shown in Listing 9 can be interpreted as "select the automatically created VNIC (net0), rename it to vphys1, and put it on the physical NIC net0." When we set the property lower-link in Listing 9, net0 is interpreted as the physical NIC, not the VNIC that we just renamed to vphys1. As mentioned earlier, that is a trifle confusing, but once you know what's going on, this should clear up the confusion.

# zonecfg -z zweb1 select anet linkname=net0; set linkname=vphys1; set lower-link=net0; end

Listing 9

Run the following command, which means "create a new VNIC, call it vweb1, and put it on top of the etherstub (webswitch1)."

# zonecfg -z zweb1 add anet; set linkname=vweb1; set lower-link=webswitch1; end

After doing this, examine the configuration by running the following command:

# zonecfg -z zweb1 export

set brand=solaris
set zonepath=/zonefs/zweb1
set autoboot=false
set autoshutdown=shutdown
set ip-type=exclusive
add anet
set linkname=vphys1
set lower-link=net0
set configure-allowed-address=true
set link-protection=mac-nospoof
set mac-address=auto
end
add anet
set linkname=vweb1
set lower-link=webswitch1
set configure-allowed-address=true
set link-protection=mac-nospoof
set mac-address=auto
end

Now we will apply the settings specified in our customized zweb1profile.xml so that when we boot the zone, it will come up fully configured. Run the following command except replace <path_to> with the path to the directory where this profile is located:

# zoneadm -z zweb1 clone -c /<path_to>/zweb1profile.xml zclone
   The following ZFS file system(s) have been created:
    rpool/zonefs/zweb1
   Progress being logged to /var/log/zones/zoneadm.20140801T013037Z.zweb1.clone
   Log saved in non-global zone as /zonefs/zweb1/root/var/log/zones/zoneadm.20140801T013037Z.zweb1.clone

And then boot the zone:

# zoneadm -z zweb1 boot   
The following ZFS file system(s) have been created:
    rpool/zonefs/zweb1
   Progress being logged to /var/log/zones/zoneadm.20140801T013037Z.zweb1.clone
   Log saved in non-global zone as /zonefs/zweb1/root/var/log/zones/zoneadm.20140801T013037Z.zweb1.clone

Log in to the console of the zone. This is useful to do as soon as the zone boots so you see any error messages sent to the console. You can't log in directly as root, so log in as the user you created when you generated the profile configuration file in Step 4. In our case, we called the user demo.

# zlogin -C zweb1
[Connected to zone 'zweb1' console]
zweb1 console login: demo   
Password: 
Oracle Corporation      SunOS 5.11      11.2    June 2014

Check that the interfaces that we created exist and have the right IP addresses (and masks!):

demo@zweb1:~$ ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
vphys1/v4         static   ok           129.200.9.2/23
vweb1/v4          static   ok           192.168.0.2/24
lo0/v6            static   ok           ::1/128

Listing 10

As you can see in Listing 10, everything looks good. As mentioned earlier, for our specific test environment we must use a 23-bit mask for the IP address that is on our external network, not a 24-bit one.

Now look at the VNICs:

demo@zweb1:~$ dladm show-link 
LINK                CLASS     MTU    STATE    OVER
vphys1              vnic      1500   up       ?
vweb1               vnic      9000   up       ?

Listing 11

Listing 11 shows we have defined two VNICs and we don't know what physical device they are on (the OVER column). That was specified in the global zone, and the non-global zone doesn't have visibility to that information. But this is good, because if we were to move this zone to another system, the receiving system would place these VNICs on an available NIC and since we aren't saying which one, we won't run into incompatibility problems. This is one reason we create everything in this zone rather than in the global zone. We pass in configuration data from the global zone, but the actual creation of the specific settings happens in the non-global zone when booting. That approach makes our zones more portable.

Escape back to the global zone using ~. (press the tilde key followed by the period key). If you use ssh to log in to the global zone, then using ~~. (pressing the tilde key twice followed by the period key) will get you out of the zweb1 zone to the global zone.

So, that is how simple it is: Set up the profile file, execute a few commands, and you have a zone with a couple of network interfaces. All that is easy to script.

Step 6: Create Three Zones Using Scripts

Let's create just the top three zones shown in Figure 4 using scripts provided in the appendix. As mentioned earlier, we do this because with these three zones we can illustrate the connectivity of the zones.

Go to the appendix and do the following:

• Using an ASCII text editor, copy the shell scripts' text and paste it into files that are named as suggested. On Oracle Solaris, gedit works great. On other systems, you want an editor that won't substitute non-ASCII quote or double-quote marks or different hyphen characters.
• In the scripts, adjust the path to the directory where the zweb1profile.xml, zrouterprofile.xml, and zvdb1profile.xml files are found.

Now create the zones. First let's get rid of the zone you just manually created.

# zoneadm -z zweb1 halt
# zoneadm -z zweb1 uninstall -F
# zonecfg -z zweb1 delete -F

Note: In the appendix, you will find a script to automate deleting a zone: cleanupzone.sh (Listing 18).

Now, we'll use the two build scripts in the appendix (Listing 15 and Listing 16) to create the three zones. So make sure the two build* scripts have been modified to include the path to the directory where you prepared the configuration profiles in Step 4.

First, create the zweb1 zone by running the following command, which depends on zweb1profile.xml being set up properly:

# sh build2vniczone.sh zweb1 vphys1 net0 vweb1 webswitch1

As soon as the command completes, run the following command to see any console messages. You might get some errors or warnings. If so, see the "Errors" section.

# zlogin -C zweb1

You don't need to log in now; you just want to be connected to the console to see the messages. So use ~. or ~~. to get back to global zone.

Then repeat the process to create the zrouter zone by running the following commands, which depend on zrouterprofile.xml being set up properly:

# sh build2vniczone.sh zrouter vweb3 webswitch1 vdb4 dbswitch1
# zlogin -C zrouter

Escape back to the global zone and then create zdb1 by running the following commands, which depend on zdb1profile.xml being set up properly:

# sh build1vniczone.sh zdb1 vdb1 dbswitch1
# zlogin -C zrouter

When you are finished, from the global zone, check what you have created:

# zoneadm list
global
zrouter
zweb1
zdb1

Listing 12

If you want more information or you are curious about what happened to zclone, it is only "installed" not "running." So it doesn't show up in the simpler invocation of zoneadm list shown in Listing 12, but it shows up using the command shown in Listing 13:

# zoneadm list -cv
  ID NAME      STATUS      PATH                BRAND      IP    
   0 global    running     /                   solaris    shared
  27 zrouter   running     /zonefs/zrouter     solaris    excl  
  33 zweb1     running     /zonefs/zweb1       solaris    excl  
  35 zdb1      running     /zonefs/zdb1        solaris    excl  
   - zclone    installed   /zonefs/zclone      solaris    excl

Listing 13

Now look at the networking:

# dladm show-link
LINK              CLASS     MTU    STATE    OVER
net0              phys      1500   up       --
net1              phys      1500   unknown  --
webswitch1        etherstub 9000   unknown  --
dbswitch1         etherstub 9000   unknown  --
zrouter/vweb3     vnic      9000   up       webswitch1
zrouter/vdb4      vnic      9000   unknown  dbswitch1
zweb1/vphys1      vnic      1500   up       net0
zweb1/vweb1       vnic      9000   up       webswitch1

Listing 14

The output shown in Listing 14 should all match the diagram in Figure 4.

Step 7: Post-Processing the Zones

Unfortunately, we cannot script everything from the global zone to have our example configuration be operational. "Operational" means that you can use ssh from every node in this network-in-a-box environment to every other node.

Our first problem is that our router zone (zrouter) will not route packets from the zdb network (192.168.0.) to other networks because, by default, zrouter is set up with forwarding set to off. The only way this routing can be accomplished is by running ipadm in the zrouter zone.

Log in to zrouter and run the following command as root:

# ipadm set-prop -p forwarding ipv4

You can check your work by running the following command:

# ipadm show-prop -p forwarding ipv4

Or just print out all the settings by running the following command, and then look at the top of the output:

# ipadm show-prop | more

The second issue stopping us is that we don't have a route from the 192.168.0 network to the 192.168.7 network. We set up our default route to the external network, but if the internal network is private to our example configuration, we need to add a route to it.

Log in to zweb1 and run the following command as root:

# route add 192.168.7.0/24 192.168.0.254

Then check the routing:

# netstat -rn

Routing Table: IPv4
  Destination    Gateway       Flags Ref Use Interface
-------------- -------------   ----- --- --- ---------
default        129.200.9.1     UG      2  45
10.132.140.0   129.200.9.2     UH      2   0 lo0
192.168.0.0    192.168.0.2     U       3  32 vweb1
192.168.7.0    192.168.0.254   UG      2 142

Step 8: Exploring the Network-In-A-Box Environment

You can now try various experiments while referring to Figure 4.

Use ssh between various zones, as follows:

• You should be able to use ssh between all zones. Use 192.168.0.2 to access zweb1 from other systems.
• You should be able to reach external hosts, via their IP address, from zweb1.
• You will not be able to access zweb1 via the 129.200.9.2 address from either zrouter or zdb1. If that is important, you can run the following command:

  # route add 129.200.9.0/23 gateway 192.168.0.2

• You will not be able to use ssh from zrouter or zdb1 to any external address unless your two internal networks (198.168.0 and 192.168.7) are known throughout your network. Otherwise, a packet can leave the network-in-a-box environment and get to an external router, but the external router will not know how to route traffic back. The same issue will cause problems if you try to use DNS on any of the nodes. Even if a route is set up and you reference only by IP address, DNS will try to map that IP address to a host name, and that will fail with a timeout after two minutes or so. So every ssh attempt results in a long delay before presenting the login prompt.

Errors

You are likely to encounter both of the following error messages during your experimenting.

The following warning can be ignored, because it is incorrectly being posted. This issue should be fixed in a future Support Repository Update (SRU) for Oracle Solaris 11.

Warning: Unable to verify add of static route on vdb1/v4
Error code = 146

If you get the following error, disable sendmail, if it is not being used within the zones. If it is being used, you need a fully qualified domain name for each zone.

sendmail[18806]: My unqualified host name (zdb1) unknown; sleeping for retry
sendmail[18806]: unable to qualify my own domain name (zdb1) -- using short name
svc.startd[17626]: network/smtp:sendmail failed: transitioned to maintenance (see 'svcs -xv' for details)

Run the following command to disable sendmail:

# svcadm disable svc:/network/smtp:sendmail

Cleaning Up

See the appendix for two scripts (Listing 18 and Listing 19) that can be used to clean up (remove) the zones and the etherstubs and restore the system to its original state. You will have to use the zfs command to remove the data set where the zones resided (but the zones will have been removed).

Conclusion

I hope this step-by-step guide will give you some ideas for future experimentation. With Oracle Solaris 11 capabilities, you can easily set up fairly complex environments. And using additional facilities not discussed here but available through the Oracle Solaris 11 repository, such as the IP Filter firewall, the quagga routing package, and the load balancer, you can address very complex networking requirements.

See Also

It is very easy to enable Apache web server in the web server zones to further show off the connectivity. See "How to Get Started Creating Oracle Solaris Zones in Oracle Solaris 11."

The Oracle Solaris documentation has a short section on the new capability of configuring multiple interfaces via editing the configuration profile XML file. See "Configuring Multiple IPv4 Interfaces."

There is also a blog about configuring multiple interfaces using the technique described above. See "Configuring Automatically Multiple NICs at Installation Time with Solaris 11.2."

About the Author

Jeff McMeekin is the last of a long line of Oracle Solaris 10 product managers and now focuses on Oracle Solaris 11.

Appendix

This appendix provides five scripts. The recommendation is to, one by one, copy a script and paste it into a file that has the suggested file name. Use gedit to create the files on Oracle Solaris 11 if you are running in a desktop environment.

Listing 15 (build1vniczone.sh) is a script for creating an Oracle Solaris Zone that has one VNIC:

#!/bin/sh

if [ $# != 3 ]
then
   echo "usage: build1vniczone.sh <zone name> <link name> <lower-link name>"
   exit 1
fi
# Hardwire the path to your profile directory, or add another 
# argument to this script.
PROFILEDIR=/export/home/crossbow/production

ZONENAME=$1
VNIC=$2
LOWERLINK=$3

zonecfg -z $ZONENAME "create -t zclone"
zonecfg -z $ZONENAME "set zonepath=/zonefs/$ZONENAME"
zonecfg -z $ZONENAME "select anet linkname=net0; set linkname=$VNIC; set lower-link=$LOWERLINK; end"
zonecfg -z $ZONENAME "export"
zoneadm -z $ZONENAME clone -c ${PROFILEDIR}/${ZONENAME}profile.xml zclone
zoneadm -z $ZONENAME boot
echo "  Finished booting, now zlogin -C $ZONENAME"

# END FILE build1vniczone.sh

Listing 15

Listing 16 (build2vniczone.sh) is a script for creating an Oracle Solaris Zone that has two VNICs:

#!/bin/sh
if [ $# != 5 ]
then
   echo "usage: build2vniczone.sh <zone name> <link1 name> <lower-link1 name>"
   echo "                        <link2 name> <lower-link2 name>"
   exit 1
fi
# Hardwire the path to your profile directory, or add another 
# argument to this script.
PROFILEDIR=/export/home/crossbow/profiles

ZONENAME=$1
VNIC1=$2
LOWERLINK1=$3
VNIC2=$4
LOWERLINK2=$5

zonecfg -z $ZONENAME "create -t zclone"
zonecfg -z $ZONENAME "set zonepath=/zonefs/$ZONENAME"
zonecfg -z $ZONENAME "select anet linkname=net0; set linkname=$VNIC1;\
                      set lower-link=$LOWERLINK1; end"
zonecfg -z $ZONENAME "add anet; set linkname=$VNIC2; \
                      set lower-link=$LOWERLINK2; end"
zonecfg -z $ZONENAME "export"
zoneadm -z $ZONENAME clone -c ${PROFILEDIR}/${ZONENAME}profile.xml zclone
zoneadm -z $ZONENAME boot
echo "  Finished booting, now zlogin -C $ZONENAME"

# END FILE build2vniczone.sh 

Listing 16

Listing 17 (createprofile.sh) is a script for configuring the profile XML files:

#!/bin/sh
#
# FILENAME: createprofile.sh
# The INFILE must be prep'ed to have the strings to
# be replaced (e.g. ZONENAME, VNIC1, etc) with the 
# values provided in this script.
# This script can handle both 1 or 2 VNIC case.  For
# 1 VNIC case the values for VNIC2 and IPADDRandMASK_2
# can be anything because they won't match a string
# in the source file.
#

if [ $# != 8 ]
then
  echo "usage: createprofile.sh "
  echo "       <profile template name> <post processing profile name>"
  echo "       <zone name>  <vnic1> " 
  echo "       <IP addr & and mask for vnic1, e.g. 192.168.0.10/24>"
  echo "       <vnic2>  <IP address & mask for vnic2>"
  echo "       <default route e.g. 192.168.0.254> "
  exit 1
fi
INFILE=$1
OUTFILE=$2
ZONENAME=$3
VNIC1=$4
IPADDRandMASK_1=$5
VNIC2=$6
IPADDRandMASK_2=$7
DEFAULTROUTE=$8

sed -e s:ZONENAME:$ZONENAME: \
    -e s:VNIC1:$VNIC1: \
    -e s:IPADDRandMASK_1:$IPADDRandMASK_1: \
    -e s:VNIC2:$VNIC2: \
    -e s:IPADDRandMASK_2:$IPADDRandMASK_2: \
    -e s:DEFAULTROUTE:$DEFAULTROUTE:  $INFILE > $OUTFILE

# END FILE createprofile.sh

Listing 17

Listing 18 (cleanupzone.sh) is a script for completely removing a zone:

#!/bin/sh 
#
# FILENAME: cleanupzone.sh 
# 
# Usage: cleanupzone.sh <zone name> 
#
# This will completely remove a zone from the system
# 
if [ $# != 1 ] 
then
      echo "Usage: cleanupzone <zone name>" 
      exit 1
fi 
echo 'zoneadm -z '$1' halt' 
zoneadm -z $1 halt 
echo 'zoneadm -z '$1' uninstall -F' 
zoneadm -z $1 uninstall -F 
echo 'zonecfg -z '$1' delete -F' 
zonecfg -z $1 delete -F
# 
# END FILE cleanupzone.sh

Listing 18

Listing 19 (cleanupdladm.sh) is a script for removing the two etherstubs:

#!/bin/sh 
# 
# FILENAME: cleanupdladm.sh 
# 
# Usage: cleanupdladm 
# 
# This will remove the etherstubs 
# 
dladm delete-etherstub webswitch1 
dladm delete-etherstub dbswitch1
#
# END FILE cleanupdladm.sh

Listing 19

Revision 1.5, 02/22/2020