Fundamental Concepts for VLAN Networks

How to understand a complex Oracle VM network infrastructure that uses bonded interfaces with VLAN tags specifically for Oracle VM 3.1 and 3.2.

Introduction

Oracle VM Server for x86 supports a wide range of options in network design. This article discusses the basic concepts relevant to Oracle VM networking. Understanding these concepts will help you make informed design choices and enable you to build a scalable, robust network architecture that has high degree of channel separation.

This article focuses on understanding a complex Oracle VM network infrastructure that uses bonded interfaces with VLAN tags specifically for Oracle VM 3.1 and 3.2. (For information on other releases, see the release-specific documentation.) Topics include Xen domains, physical hardware choices including network bonding, logical Oracle VM networks and network channels, and VLAN segments and groups.

Note: This article does not specifically step through planning and implementing Oracle VM networks. A future article, dedicated to designing a complex Oracle VM network based on the concepts covered in this article, is planned.

About for Oracle VM Server for X86

Oracle VM Server for x86 supports a wide range of options in network design, varying in complexity from a single network to configurations that include network bonds, VLANS, bridges, and multiple networks connecting the Oracle VM servers and guests. You can create separate networks to isolate traffic, or you can configure a single network for multiple roles. Network design depends on many factors, including the number and type of network interfaces, reliability and performance goals, the number of Oracle VM servers and guests, and the anticipated workload.

The Oracle VM Manager GUI presents four different ways to create an Oracle VM network (see Figure 1):

• Bonds and ports
• VLANs
• Both bond/ports and VLANS
• A local network

This article focuses the second option, designing a complex Oracle VM network infrastructure using only VLANs, and it steps through the concepts needed to create a robust network infrastructure for your Oracle VM servers and guests. Using VLANs allows you to create networking schemes that go far beyond the number of physical network devices installed in your server.

Note: Many networking internals are exposed throughout this article; this information is used simply to help bring clarity to concepts—using any information about internals for the purposes of bypassing Oracle VM Manager or performing work on the command line is not supported or recommended under any circumstance. Using the command line to "save time" will only result in costing more time and will seriously compromise the integrity and reliability of your Oracle VM implementation.

Understanding Xen Domains

Oracle VM Server incorporates an open source Xen hypervisor component, which has been customized and optimized to integrate into the larger, Oracle-developed virtualization server. For this article, you need a basic understanding of Xen domains, since the terms Dom0 and DomU are used while explaining the relationships among various network components. The following explanation is greatly oversimplified, but sufficient for this article.

There are only two types of domains that are supported by the Xen hypervisor:

• Oracle VM Server: Dom0. In this article, the terms Dom0 and Oracle VM Server are used interchangeably. Dom0 is the management domain, which manages one or more DomU virtual machines (VMs). All network traffic flows to and from Oracle VM guests (DomU) through the network devices created by Dom0. These logical network devices include ports such as eth0 and eth1, port bonds, VLAN ports, Xen bridges, and netback devices that shuttle network traffic between the Xen bridges on Dom0 and DomU.
• Oracle VM guests: DomU. In this article, the terms DomU and Oracle VM guest are used interchangeably. Every time an Oracle VM server (Dom0) starts an Oracle VM guest (DomU), the guest is assigned an arbitrary domain ID. Domain IDs are very transient: The next time the guest is stopped and then started again, it assigned a completely new arbitrary domain ID. The U in DomU is simply a placeholder where you substitute the domain ID of a running Oracle VM guest. With this nomenclature, we can refer to particular guests by changing DomU to some fictitious name (such as Dom22 or Dom23) to illustrate an idea.

Although neither of the above descriptions is technically complete, this explanation will suffice for understanding the networking concepts in this article.

Understanding Hardware Choices

The first step in planning an Oracle VM network infrastructure is to determine the physical network environment:

• Network switches. To facilitate troubleshooting and planning, you need to understand which Ethernet switches your servers are connected to. You should know the negotiation setting, speed, duplex, and VLAN assignments for each port. Your network administrator should be able to provide all this information. If you are using Cisco IOS, the show int status command is very useful for gathering this information from each switch.
• Network devices. For each of your Oracle VM servers, determine the number and type of network interfaces installed. Once you have Oracle VM Server installed, you should also determine which physical port is related to which Ethernet device (eth0, eth1, and so on). You should also know exactly which Ethernet switch and port each server port is connected to—this will save countless, frustrating hours of troubleshooting if something is not working correctly.
• Backplane layout. All servers in a single server pool (if not all pools) should have I/O and network cards in the same slot positions. Planning identical hardware configurations for all servers, although not required, is highly recommended to simplify configuration and reduce both implementation time and the chance for human error.

Understanding Oracle VM Networks

Figure 2 shows a screenshot depicting the Oracle VM network objects used as examples throughout this article.

Each line in the screenshot shows a single Oracle VM network that represents two things:

• A subnet/VLAN. If you have only one subnet available to your Oracle VM servers, then you will have only one Oracle VM network. However, our example uses multiple VLAN segments, and we have ten subnets available for each server pool in a single data center (pool1, pool2, and pool3). That is, we have a total of thirty subnets that we are allocating across three server pools to perform various functions for us.
• A network channel. A network channel describes how a given subnet will be used to support our Oracle VM environment. Oracle VM uses the term network channel to describe network roles. Network channels are explained in greater detail in the later section "Understanding Oracle VM Network Channels."

Take a moment to study the descriptions for each network shown in Figure 2 to get an understanding of how each Oracle VM network is used. Notice on the fourth line that the name of the network object helps describe how it is used. The name is simply a text field containing anything you like. Oracle recommends assigning a name to help make the usage clear. For example, "Guest DataCtr1 POOL1 PUB bond1" indicates that this network is used as the primary public interface for accessing any Oracle VM guests in pool1, and it will be found assigned to bond1 on all servers in the pool.

Continuing along the same row in Figure 2, notice that the assigned virtual machine channel indicates IP addresses can be assigned only on the VM guests (DomU), not on Dom0. Further along on the same row, the description text field is used to tell us that this network should be assigned as eth0 on each guest, and it is associated with VLAN ID 200 on bond1. Again, both text fields (name and description) can contain anything you want or nothing at all; but the more information you provide, the easier maintenance and troubleshooting will be.

You create Oracle VM network objects by assigning a single VLAN segment from a VLAN group. At this point, you give the network an easy-to-understand name and perhaps a description to help explain further what role the network fulfills. The VLAN segment defines which subnet and bond the network traffic will flow over.

In our example, an Oracle VM network object is a construct using a combination of other network objects as attributes:

• Ethernet ports. These are Ethernet ports such as eth0 and eth1. The Ethernet ports can be used to create bonded interfaces or you can leave them unbonded—the choice is up to you.
• Bonds. Up to four Ethernet ports can be aggregated together to create a single logical interface. (Prior to Oracle VM 3.2, only two Ethernet ports could be combined into a single bond.) Bond0 is created during installation, and only one Ethernet port is added to bond0 initially. Bond0 can remain with a single port, or you can add up to three additional ports. For our example, we will aggregate six Ethernet ports into three bonded interfaces named bond0, bond1, and bond2. Bonds are explained in greater detail in the later section "Understanding Bonding."
• VLAN segment. A VLAN segment is the VLAN ID for a particular subnet that your network administrator has configured on the Ethernet switches your server is connected to. VLAN segments and groups are explained in greater detail in the later section "Understanding VLAN Segments."
• VLAN group. The VLAN group contains all the VLAN segments that you want associated with a particular bond (or port). Multiple bonds (or ports) can be assigned to a VLAN group. However, once a bond or port is assigned to a VLAN group, the bond or port can no longer be assigned to any other VLAN groups or networks: the VLAN group owns the bond or port at that point forming a one-to-one relationship. This means you cannot assign more than one VLAN group to the same bond or port.
• Network channels. Network channels further define how a particular Oracle VM network will be used. Network channels are explained in greater detail in the later section "Understanding Oracle VM Network Channels."

Understanding Bonding

Figure 3 shows a screenshot of network bond ports in an Oracle VM configuration. Bonding is simply aggregating or associating more than one Ethernet port into a single logical network interface. By aggregating ports into a single interface, you can achieve high availability by eliminating single points of failure for a given subnet: network traffic will still flow across all remaining Ethernet ports if a single port fails in a bonded interface. The effectiveness of your high availability solution relies heavily upon your data center infrastructure team ensuring the Ethernet ports on the Oracle VM servers are physically connected to redundant switch infrastructures.

Oracle VM 3 provides three different types of bonding modes that can be used to aggregate multiple network devices (ports) into a single bond. Up to four ports are supported per bonded interface in Oracle VM 3.2; Oracle VM 3.1 and previous releases support a maximum of two ports per bond. All network devices/ports that are part of a bond must be the same type and speed.

All of the following modes are highly available: in the event of a network port failure, network traffic will continue to flow across all remaining devices/ports. Two of the three bonding modes also provide aggregated bandwidth, effectively increasing the total bandwidth of a bonded interface by the number of Ethernet ports that are part of a bond.

• Active/backup. Mode 1 can be configured within Oracle VM Manager without involving a network administrator. As the name implies, only one network port (such as eth0 ) is used for network traffic while a second port (such as eth1) takes over only if the first port fails. Active/backup mode supports VLAN segments with VM traffic.
• Adaptive load balancing. Mode 6 can be configured within Oracle VM Manager without involving a network administrator. This mode will send network packets over each network port, alternating the device it uses for each packet; this mode effectively simulates link aggregation. So, if you have two 1-gigabit network ports in a single bond, it will seem like the two cards are aggregated together to provide 2 gigabits of throughput, but in reality this is not actually the case. Be careful with this mode, however, since it does not support VLAN segments with VM traffic.
• Dynamic link aggregation. Mode 4 must first be configured on the Ethernet switch infrastructure by your network administrator before it can be configured in Oracle VM Manager. Network administrators will use Cisco's Link Aggregation Control Protocol (LACP) to configure Ethernet channel groups. Dynamic link aggregation supports VLAN segments with VM traffic.

Figure 4 shows the network device that is created on each Oracle VM server when you create a bond in Oracle VM Manager. Using Oracle VM nomenclature, bonds aggregate one to four Ethernet ports into a bond port.

Notice that Figure 5 shows bond0 using active/backup mode. It is important to also note that we do not assign any virtual machine channel roles to bond0 at any point. Do not use adaptive load balancing if there is any chance you want to assign the virtual machine channel for Oracle VM guest traffic on bond0 at some point in the future. If you are going to use bond0 for Oracle VM guest traffic, then use either active/backup or dynamic link aggregation as the bonding mode for bond0.

Configuring LACP on the network infrastructure is normally complex and time consuming for network administrators—make sure you give your network administrators plenty of lead time to configure and validate LACP. If you do not have the ability to work with your network administration team, consider using active/backup or adaptive load balancing for your bonding mode, keeping in mind the limitation that adaptive load balancing has with VLANs and Oracle VM guest bridges.

Understanding VLAN Segments

Figure 6 shows a screenshot of VLAN segments in an Oracle VM configuration.

Oracle VM uses existing VLAN segments; it does not create VLAN segments. The VLAN segments (tags) must be configured on the physical Ethernet switches before they can be used as part of the Oracle VM network infrastructure. Your network administrator will tell you which VLAN IDs are assigned to which ports on a switch. You then need to ensure the network ports on your servers are connected to the correct switch ports that are configured with the VLAN IDs you expect to access across any given network port.

Prior to the VLAN specification, a single network port was able to shuttle traffic across a single subnet. So, if you had four non-bonded network ports on a server, you could have access to a total of four different subnets. The purpose of the VLAN specification is to allow a single network port to access multiple subnets across a single wire connected to a single switch port. Therefore, by using multiple VLANs assigned to a single switch port, you can access multiple subnets on a single network port. This means the same server with four physical network ports can access more than four subnets.

Any number of VLAN segments or trunks can be associated with a single port or bond. However, there is a practical limit to the amount of network traffic based on the potential throughput of the port or bond. For example, if you assign four VLANs to a single 1-gigabit Ethernet (GbE) port and the anticipated network load on each VLAN is 800 megabits, you are most likely going to have a network performance problem because the port might be oversubscribed: you could potentially have 3.2 gigabits (3200 megabits) of traffic on a 1-GbE port.

Each VLAN ID represents a separate subnet (or broadcast domain) with its own IP range, subnet mask, and gateway. A network device is created on each Oracle VM server when a VLAN group containing one or more VLAN segments is assigned to a network port, such as bond1 shown in Figure 7.

Understanding VLAN Groups

VLAN groups are simply containers for any number of VLAN segments that you want associated with a particular port or bond on each server. At the time of this writing, there is a one-to-one relationship between any given bond or port and a VLAN group: once you have assigned a VLAN group to a port or bond, the port or bond cannot be used for anything else.

VLAN groups are a required component when creating Oracle VM networks using VLANs, whether you are implementing a simple or complex network infrastructure using VLANs with ports or bonds. VLAN groups are the key to success for designing complex networks when a variety of server pools are being controlled by a single Oracle VM Manager. The planning of VLAN groups is where you should spend a majority of your time, since you will not be able to create and deploy complex Oracle VM networks if you fail to plan VLAN groups correctly—so spending the time to really think this through can save you days or weeks of frustration down the road.

In Oracle VM, VLAN groups allow you to compartmentalize small and large numbers of subnets into logical groupings that make sense for your scenario. For example, you might have 300 VLAN segments that are used for various reasons, but the guests in server pool1 need only 6 of the 300 subnets, while guests in server pool2 need 22 of the 300 subnets. VLAN groups allow you to customize which VLANs the Oracle VM servers in each server pool will have access to, without cluttering up every server with hundreds of extraneous VLANs they might never use.

Although only an example of one way to design VLAN groups, Figure 8 illustrates that you can create VLAN groups specific to particular server pools in different data centers. Just because the servers in each server pool have a bond0, bond1, and bond2 doesn't mean you are limited to only a single VLAN group for each bond—each server pool can have a completely different set of VLAN segments for each bond, depending on what the servers in each pool need. (Please don't let the above example limit your imagination—there are many ways to approach the problem, so design a scheme that works for the requirements in your data center.)

Create VLANs groups before you attempt to create Oracle VM networks. Although there are several different opportunities to add IP addresses through a variety of network wizards, you should get in the habit of assigning all IP addresses that are relevant to Dom0 at the time you create each VLAN group, not when you create the Oracle VM networks.

Figure 9 briefly shows how VLAN groups and segments are used to create each Oracle VM network. In this example, we are creating an Oracle VM network named "OVM DataCtr1 POOL1 bond2.902" that will be assigned a network role of Live Migration and will be dedicated to live migration of Oracle VM guests between Oracle VM Servers.

Figure 10 illustrates the logical association of the VLAN segments in each VLAN group to each bond from the perspective of an Oracle VM server in pool1. It is important to note that VLAN segments in each VLAN group are available only to the bond that is assigned to a particular VLAN group.

Understanding Oracle VM Network Channels

Let's explore what network channels do in the context of Oracle VM network objects. When you create an Oracle VM network using the Oracle VM Manager GUI, you assign one or more of the following channels: server management, cluster heartbeat, live migrate, storage, or virtual machine (see Figure 11).

Selecting various channels triggers different behaviors from both Oracle VM Manager and the Oracle VM servers. The first three channels (server management, cluster heartbeat, and live migrate) pertain only to server pool management and configure various features on Dom0; these channels have absolutely nothing to do with Oracle VM guests (DomU). The fourth channel (storage) shows storage that currently provides no functionality in the product. The fifth channel (virtual machine) is strictly related to Oracle VM guests and has nothing to do with the Oracle VM servers (Dom0).

• [Dom0] Server Management. Used by Oracle VM Manager to control the physical Oracle VM servers within any given server pool (required). Selecting this box tells Oracle VM Manager to communicate with the Oracle VM master server in each pool using this subnet. This channel is automatically configured during the installation of Oracle VM Server. Never attempt to change this network channel after installation for any reason, not even temporarily.
• [Dom0] Cluster Heartbeat. Used by Oracle Cluster File System 2 to send heartbeat messages between the Oracle VM servers in a server pool to verify that the Oracle VM servers are present and running (required). When we talk about cluster heartbeat for Oracle VM, we are referring to the built-in clustering features of Oracle Cluster File System 2. Selecting this box when creating a network doesn't do anything more than instruct the agent on each Oracle VM Server to update the Oracle Cluster File System 2 cluster configuration file (/etc/ocfs2/cluster.conf) file with the IP address and host name of each server in the server pool.

A natural inclination for high availability is to assign this role to multiple Oracle VM networks in case one fails, with the assumption that the heartbeat will continue running on any remaining networks within the same server pool. However, this is incorrect since OCFS2 supports only a single IP address/host name for each server in the cluster.conf file. So, assigning this role to multiple networks within the same server pool doesn't actually accomplish anything; each server pool can have only one network with the cluster heartbeat role.

The network assigned the role for cluster heartbeat should be separate from networks with high load, such as live-migrate or Oracle Real Application Clusters (Oracle RAC) private traffic. Each server pool can share a common network for heartbeat, but this is not scalable for Oracle VM environments with many server pools containing large numbers of Oracle VM servers. Although not a requirement, an Oracle-recommended best practice is to have a completely separate VLAN/subnet dedicated to cluster heartbeat for each server pool. Of course, this might be a moot point if all you have is a single bond0 using the same physical interfaces as all your other servers.

• [Dom0] Live Migrate. Used by Oracle VM when migrating VMs from one Oracle VM server to another in the same server pool (required). Oracle VM uses the Xen migrate command to live migrate an Oracle VM guest from one server to another in a server pool with zero downtime to any applications hosted by the guest operating system. Selecting this box simply tells the agent on the Oracle VM server to use the IP address associated with the destination server on this particular network when executing the command xm migrate -l . This network should be separate from networks with high load, such as cluster heartbeat.
• [Dom0/DomU] Storage. The storage channel doesn't control anything about which subnet storage traffic is allowed to flow over. The storage channel simply tells Oracle VM Manager to allow you to add IP addresses to ports on Dom0. This is needed only if you create on Dom0 an Oracle VM network that you want NFS or iSCSI traffic to flow over and you have no reason to assign any of the other network channels. Basically, assign this role to an Oracle VM network only if you need to add an IP addresses to a port/interface on Dom0 but are not assigning the server management, cluster heartbeat, or live migrate channels.
• [DomU] Virtual Machine. The virtual machine channel is used for communication among the Oracle VM guests in the server pool and also between Oracle VM guests and external networks. Selecting this box when creating an Oracle VM network instructs the agent on all Oracle VM servers to create a Xen bridge network device. In our example, we do not assign a virtual machine channel to any of the management networks because we are seeking complete channel separation, but there is nothing that prevents you from adding the virtual machine channel to a network used for server management, cluster heartbeat, or live-migrate if your particular situation requires this configuration.

Selecting the virtual machine channel in Oracle VM Manager instructs the agent on each Oracle VM Server to create a Xen bridge that is associated with the port, bond, or VLAN segment. A bridge acts as a network switch on each Oracle VM server and shuttles network traffic between DomU and the rest of the world. A bridge will be created for any Oracle VM network you create that has the virtual machine channel selected; each bridge has a unique randomly generated network ID that can be seen in the Networking tab in Oracle VM Manager. Figure 12 shows the relationship between the VLAN interface (bond1.200) and the netback driver for Oracle VM guests.

The server management, cluster heartbeat, and live migration network roles are required for each Oracle VM server. The storage network role is not fully implemented at the time of this writing. Each Oracle VM environment will also have one or more virtual machine networks.

Figure 13 shows the network channels used by Oracle VM and illustrates the varying scope for each channel type. The point of the illustration is simply to show which Oracle VM components need to communicate with each other and with the outside world. The management traffic for Oracle VM (server management, cluster heartbeat, and live-migrate) can be restricted to small, non-routed subnets dedicated to each function. For example, if your server pool is contained in a single blade chassis, the cluster heartbeat and live migrate channels can be connected to a Ethernet switch that is configured only for intra-chassis communication—there is no reason to connect all the way to a switch or router outside of the chassis.

The server management channel includes communication between Oracle VM Manager and the Oracle VM servers in the server pool, but not the guests. Both the cluster heartbeat and live migrate channel carry traffic among only the Oracle VM servers, but not Oracle VM Manager. For these channels, traffic can be on private, non-routed subnets—the traffic doesn't require access to routers. The traffic requires access to only the switches that connect the servers and manager, because communication flows only between the servers in the pool.

In contrast, a virtual machine channel includes traffic between Oracle VM guests as well as communication with external public networks. When you use the command-line interface to explore the networks configured by Oracle VM Manager, commands executed on the Oracle VM server will show network devices for all channel types. In contrast, commands executed on an Oracle VM guest will show only network devices for networks that are assigned the virtual machine channel.

Network channels are assigned when you create an Oracle VM network or you are changing network infrastructure at any point in the future.

Final Thoughts

Oracle VM Server for x86 supports a wide range of options for network design. Networks can vary in complexity from the simplest single-network configuration to configurations that include network bonds, VLANS, bridges, and multiple networks connecting the Oracle VM servers and guests. Network design depends on many factors, including the number and type of network interfaces, reliability and performance goals, the number of Oracle VM servers and guests, and the anticipated workload.

Understanding the basic concepts of Oracle VM networking is essential for designing a complex Oracle VM network infrastructure with a high degree of channel separation for a more scalable architecture.

See Also

The following resources are available for Oracle VM Server for x86:

• Oracle VM documentation, including the Oracle VM Installation and Upgrade Guide and Oracle VM User's Guide: http://docs.oracle.com/en/virtualization/oracle-vm/index.html
• Oracle VM portal: http://www.oracle.com/virtualization/technologies/vm/
• Oracle Virtualization Website: http://www.oracle.com/virtualization
• Oracle's virtualization blog: http://blogs.oracle.com/virtualization/
• "Looking 'Under the Hood' at Networking in Oracle VM Server for x86": http://www.oracle.com/technical-resources/articles/it-infrastructure/admin-networking-ovm-x86.html

About the Authors

Gregory King joined the Oracle VM product management team as a senior best practices technical consultant specializing in Oracle VM Server. Greg has a significant amount of hands-on experience with high availability, networking, storage, and storage networking, including SAN boot technology and server automation in production data centers, since 1993. His primary responsibility with the team is to develop and document solutions, including best practices, as well as foster a deeper understanding of the inner workings of Oracle VM with the goal of helping people become more successful with the product.

Don Smerker has been in the IT industry for more than twenty years, joining Oracle during the acquisition of BEA Systems, where he was part of the network operations team designing and managing all aspects of the network infrastructure. After joining Oracle, Don was part of Oracle's GIT networking organization responsible for day-to-day management of the network infrastructure for office spaces and off-production data centers. Don currently works with Oracle's GSS Network Operations team on key initiatives restructuring, reorganizing, and consolidating network infrastructure with some continued day-to-day management aspects of network operations.