Exploring the Cisco UCS Server Environment

The Cisco UCS uses hardware abstraction to provide flexibility, better utilization of resources, and redundancy for the servers. The abstraction is achieved by decoupling the server’s hardware from the server’s configuration that defines its behavior. To put it another way, the Cisco UCS cluster is built as a physical infrastructure. This is the understructure, just like the road infrastructure. There are multiple roads to take you from point A to point B. Which road you take depends on the requirements for your trip (how fast do you want to get there? are there tolls on the road?), on the availability, and on the traffic of the roads on a specific route. But these conditions change for your car and trip; the infrastructure is not affected by the parameters of your trip. Here it is similar—the physical infrastructure is built and available after the physical components of the Cisco UCS are installed (their cabling and the initial configuration) and the cluster is brought up. Of course, this includes the configuration of connectivity to the LAN and SAN of the data center. But, to explain this, first you need to get acquainted with the physical infrastructure and some of the connectivity options when the UCS cluster is built.

In order to keep things clear and simple, our explanation will focus on a Cisco UCS cluster that consists of a pair of Fabric Interconnects, a blade chassis, and B-series servers in the chassis.

A pair of Fabric Interconnects is a requirement for a Cisco UCS cluster. A topology with a single Fabric Interconnect is allowed for lab and test purposes, as it does not provide redundancy. The two Fabric Interconnects must be connected to each other only using the dedicated cluster ports, which are marked with L1 and L2 (L1 connects to L1, and L2 connects to L2). This is all you must do for the cluster link—connect the cluster port. The system will automatically detect the link and configure a port channel using the two physical links. This happens transparently.

Additionally, there is the management plane. On each Fabric Interconnect is a dedicated Ethernet port for management communication. They must be connected to the management switches. Each port is assigned an IP address from the management network. There is one additional IP address from the management network, called the cluster IP address. It is assigned to the active instance of the Cisco UCSM and floats between the Fabric Interconnects, depending on which one of them is running the active instance. The management plane has a separate physical path of communication from the data plane. Further discussion in this chapter will focus on the data plane communication, as it is important for understanding the hardware abstraction in the Cisco UCS.

When it comes to the data plane connectivity of the Cisco UCS, you have external connectivity to the LAN and SAN infrastructures of the data center and internal connectivity to the system, starting from the Fabric Interconnects and going down to the servers.

The external connectivity to the LAN infrastructure is created by linking the Ethernet ports of the Fabric Interconnects to the upstream LAN switches. Based on best practices, and in order to achieve redundancy, each Fabric Interconnect can be connected to a pair of upstream LAN switches. Technologies such as port channels or vPCs can be used for adding additional redundancy and load balancing. For the Cisco UCSM to know that through these specific Ethernet ports the Cisco UCS is connected to the LAN infrastructure, the ports need to be configured as such. This happens by selecting the port on the Fabric Interconnect and assigning it the Uplink Port role.

Creating and configuring the connectivity to the SAN infrastructure is similar, but there are some differences. Again, you have to select ports to connect to the SAN. This time there are some requirements and some options. The major choice that needs to be made, depending on the design of your data center, is which storage communication protocol to use. The options are iSCSI, NFS, CIFS/SMB, FC, and FCoE. For the iSCSI, NFS, and CIFS/SMB options, there is no need for any additional upstream connectivity other than the LAN connectivity, as these protocols are using Ethernet and IP connectivity to carry the storage communication.

For the FCoE connectivity, the Ethernet uplink ports need to be configured as FCoE Uplink ports or as Unified ports, depending on whether you will use dedicated Ethernet links for the storage communication or you will use the same Ethernet links for both Ethernet and FCoE traffic.

In both cases, you must consider the SAN design best practices, which require each server to be connected to two separate physical SAN infrastructures. As all the server’s data plane communication goes through the Fabric Interconnects, this means that each of them can be connected to only one SAN, as they will become an extension of the SANs down to the server.

The same design considerations apply if you want to connect the Cisco UCS to the SANs using native Fibre Channel connectivity. For this, you must use dedicated FC ports. Once again, each Fabric Interconnect connects to a single SAN (no cross-connects like in LAN connectivity). As discussed Chapter 11, “Fibre Channel Protocol Fundamentals,” the redundancy is achieved in a different way than in the LAN infrastructure. The FC ports must be assigned the Fiber Channel Uplink ports role.

Figure 17-14 illustrates the internal and external physical connectivity of Cisco UCS.

Figure 17-14 Cisco UCS Server Environment – Physical Connectivity

Once external connectivity, or upstream connectivity to the other elements of the data center, is created, it is important to understand how the internal connectivity is realized so that you know what is available to the servers to consume.

As already mentioned, the servers’ communication always goes through the Fabric Interconnects. Here we will explain the physical internal connectivity of the Cisco UCS starting with the Fabric Interconnects and moving down to the servers.

In each Cisco UCS 5108 blade chassis are two FEXs that connect to the Fabric Interconnects externally, thus securing the blade chassis’ outside connectivity. Each FEX connects to only one of the Fabric Interconnects. This allows for the separation of the physical infrastructure to be maintained down to the level of the physical server. This is Ethernet connectivity, as the Ethernet links are shared to carry the management and control communication, the Ethernet data communication of the servers, as well as their storage communication using the FCoE protocol. That’s why these links operate at speeds of 10, 25, and 40Gbps.

The number of links between the FEX and the Fabric Interconnect define the total bandwidth of the connectivity of the chassis using that path. This is the available communication capacity to the servers in that chassis for this path. The other FEX is connected to the other Fabric Interconnect in the same way. This ensures the blade chassis has two paths of communication: one through Fabric Interconnect A and the other through Fabric Interconnect B. Both paths are active and available.

Another important point is that the FEXs do not perform Layer 2 switching for the data plane communication. This results in one hop less of processing the server communications. This makes transmission faster, and it decreases latency. This is possible because, instead of the standard Layer 2 switching, inside the FEX is a mapping between the external ports and the internal ports.

The FEX is connected to the midplane of the blade chassis. The midplane is a passive component that provides multiple electrical lanes for the signal, hard wiring the different components as required. Also connected to the midplane are the blade servers when they are installed in the chassis; this is done through a special blade connector. Therefore, the midplane provides the internal physical connectivity at the lowest physical layer. Think of the midplane as the physical network cabling. This physical connectivity is utilized by the internal ports on the FEX, from one side, and the mezzanine adapter of the server. The FEX has dedicated internal ports for each server slot in the chassis. Once installed in the blade chassis, these ports are connected to the midplane and are available to the server slots. How much of the internal FEX data connectivity will be utilized by a blade server depends on two things:

The supported connectivity by the mezzanine adapter in the server
The number of mezzanine adapters in the servers

The mezzanine adapter in the blade server is installed in a specific slot, which provides for the physical connectivity in both directions:

External: The mezzanine adapter, through the mezzanine slot and the blade connector, is connected to lanes from the midplane. For each mezzanine slot, half of the lanes of the midplane are hard-wired to one of the FEXs, and the other half lead to the other FEX. If we look at the Cisco UCS 1380 mezzanine adapter, shown in Figure 17-15, we see that it has two 40-Gbps ports, as each connects to one of the available paths through the midplane to one of the FEXs. Each of the 40-Gbps ports can be divided into four 10-Gbps ports. Depending on the capabilities of the installed FEXs, the server can have up to 40-Gbps available bandwidth for each path.

Figure 17-15 Cisco UCS Mezzanine Adapter, VIC 1380

Internal: Again, through the mezzanine slot, the adapter is connected to the PCIe bus of the server. Through the PCIe bus, the OS is capable of seeing and utilizing the vHBAs and the vNICs on the mezzanine adapter. The Cisco virtual interface cards utilize a programmable silicon that allows you to program (or create) multiple NICs and HBAs for the purposes of the OS on the server and for providing redundant connectivity and load balancing. These logical adapters are called vNICs and vHBAs, but these are not the vNICs and vHBAs from the VMware ESXi hypervisor. Take note that although the terminology is the same, these are two different vNICs and vHBAs. Let’s try to clarify it a bit more: as you know, the VMware ESXi creates a virtual image of the physical hardware, and after that the virtualized resources are utilized to create the isolated spaces, which are called virtual machines. In the case of networking, the ESXi hypervisor creates a virtual image of the network interface card that it sees on the server and names it a vmNIC. Afterward, when a VM is created, a logical network adaptor is created, named vNIC, which connects to a virtual switch, which in turn is connected to a vmNIC for the external connectivity of the ESXi host and the VMs. In the case of using a Cisco VIC, the Cisco vNIC will be seen as a “physical” network adaptor by the ESXi and will be created. Therefore, the Cisco VIC vNIC is equal to a VMware ESXi vmNIC. This is important for you to understand to avoid further confusion.

To summarize, starting with the Cisco VIC in the server and going through the midplane and the two FEXs, and reaching the two Fabric Interconnects, two communication paths are available for each server. These paths are physically separated, and their bandwidth depends on the capabilities of the hardware used, such as FEXs and mezzanine adapters. However, this is the physical infrastructure for the communication of each server. What and how much of it a server will utilize depends on how you configure it in the Cisco UCS Manager by using a service profile. Before we can explain the service profiles, which allow us to abstract the hardware resources of the Cisco UCS, you will need to understand the challenge of the static hardware servers.