*Reposted with permission from Oracle’s Networking Blog and Neeraj Gupta
Virtualization is always a heavily talked subject among infrastructure owners. Someone may ask why ? Often times, at any given time, the capacity of hardware equipment used to deploy services may be more than the services can actually utilize them. Data center real estate and power is expensive.
So, what do we want to do ? Consolidation.
Hmm.. sure but what about security ? Someone will not like the fact that they are running on a shared set of resources with everything exposed.
Now, what do we want here ? Isolation.
These two keywords - Consolidation and Isolation, may appear to be contradicting to each other but together, they form the basis of Virtualization in computing environments.
In networking world, virtualization is achieved when you can consolidate multiple data flows over a given media and yet maintain isolation.
I have been hearing people asking these question quite a lot of times.
Whats with these VLANs ? How many should I create ?
What are InfiniBand partitions ? How do I implement them ?
Lets twist these questions around and ask differently.
How many different local area networks we need to participate in ?
Do we need security amongst these networks ?
In modern day computing environment, almost all hosts and especially servers are part of more than one network. There is a term for this - Multi Homing. Lets elaborate a little bit on this. The meaning is quite literal. It is about participating in different networks at the same time. Lets start with one network. What do you need here ? Simple enough. One network port with one IP address and subnet mask. You will add a gateway address just in case your destinations are out there in the bigger network beyond your local area. Okay, things grow and your requirements may expand too. Add another network port, another IP address, mask and gateway etc. But can you scale with this model beyond a certain limit ? No. Very soon you will realize that you are hitting a physical limitation to expand. And nobody would like to spend more money on hardware and then manage more things on the data center floor. Those who have seen the cabling know what I am talking about.
So what do we need to do here ? Consolidation. How do we do that ? Lets see... and keep in mind the first three layers of OSI because I will be referring to them going forward. As soon as we virtualize at a layer, the layers above it will inherit.
Virtualization at Layer 1 (Hardware)
Even if this is for a simple explanation, I would say we do have some consolidation happening at hardware or layer 1. Haven't you seen network interface cards with more then one network ports ? How about those dual band wireless routers ? At this layer, this is the kind of consolidation we have and isolation is pretty much built in. Instead of having two single port network cards, we may use dual-port or even quad-port cards. Different ports on the same card will have their own hardware address and physical path to external world.
|Interfaces from NIC #1|| Interfaces from NIC #2 ||Interfaces from NIC #3||Interfaces from NIC #4|
|Ethernet||eth0, eth1||eth2, eth3||eth4, eth5||eth6, eth7, eth8, eth9|
|InfiniBand||ib0, ib1||ib2, ib3||ib4, ib5||ib6, ib7|
As you can see that each individual network interface card provides a unique interface to be used at layer and layer 3. The consolidation shown here is only at Network Card level and this is purely to build up some conversation here for understanding.
The fun starts here . In this section, we will stay on same physical media or lets say same network port which has a fixed hardware address. In Ethernet, this is known as MAC and in InfiniBand, it is Port GUID or LID under subnet management. We can consolidate multiple streams at layer 2 as the upper layer 3 can participate in multiple IP subnets but how do we achieve islolation here ? That will be the key for virtualization here.
Ethernet world implements Virtual LAN based on 8021.Q specifications
InfiniBand implements Partition Keys (PKeys) based on IBTA specifications
Basically, the idea is to allow a unique ID for each stream consolidated at link layer 2 and present the new interface to upper layer with isolation. The overall result is network virtualization at layer 2.
|Interfaces from NIC #1|| Layer 2 Isolation Unique ID || Interface Presented to Layer 3 |
|Ethernet||eth0, eth1||VLAN=20||eth0.20, eth1.20|
|InfiniBand||ib0, ib1||PKey=0x8001||ib0.8001, ib1.8001|
In this approach, when you analyse the packets traversing on the wire, there will be unique fields set at layer 2 to isolate packets. Layer 3 will also have its own source and destination IP subnet information.
Virtualization at Layer 3 (Network)
This is the layer where we use IP addresses at end points and I will refer to Linux in my examples. Assumption here is that we have the same network interface with same hardware address and no virtualization at layer 2. So, how do we do this at layer 3 ? IP aliasing with unique subnets. Lets say our interface is eth0. Several operating systems, including Linux allows us to add more virtual interfaces with namig format like eth[n]:[y]. Here 'n' is our interface instance and 'y' will be a virtual instance on top of it. So, we can have several virtual layer 3 instances participating in their own IP subnets. What did we do here ? Consolidated data streams and also isolated via unique subneting.
|Interfaces from NIC #1|| IP Aliases or Virtual Interfaces |
|Ethernet||eth0, eth1||eth0:1, eth0:2, eth1:1, eth1:2|
|InfiniBand||ib0, ib1||ib0:1, ib0:2, ib1:1, ib1:2|
In this approach, when you analyse the packets traversing on the wire, the only differences will be at layer 3 in source and destination IP subnets. There will be no isolation at layer 2.
What works best ?
We just saw how virtualization can be done at three lower layers of OSI model. Lets evaluate and recap.
Layer 1: Requires more hardware and becomes un-manageable. Many people will just debate that this is not virtualization. I agree and my purpose was to give you an idea on what happens at each layer.
Layer 2: This is the closest layer to hardware and once virtualized, the upper layers inherits the environment. Provides best confidence level in terms of security and isolation.
Layer 3: Virtualization here does the job but confidence level falls due to unprotected layer 2.
The choice is yours to make and it all depends on how you design your infrastructure. Advantages of virtualization at layer 2 seems to outweigh other options. The technologies at hand here are VLANs for Ethernet and Partition Keys for InfniBand.
In my upcoming blogs, I will give you more insight on how Partitions and VLANs are actually implemented using products at hand and how do we consolidate services while maintaining isolation to achieve virtualization at network layers.
*About the author: Neeraj Gupta joined Oracle as part of Sun Microsystems aqusition, where he spent last 11 years specializing in InfiniBand, Ethernet, Security, HA and Telecom Computing Platforms. Prior to joining Sun, Neeraj spent 5 years in Telecom industry focusing on Internet Services and GSM Cellular Networks. Currently Neeraj is part of Oracle’s Engineered Systems team focusing on Networking and Maximum Availability Architecture.