Deployments over InfiniBand Infrastructure

*Reposted with permission from Oracle’s Networking Blog and Neeraj Gupta

I have been talking to many people who are working hard towards migrating an existing Ethernet based infrastructure or starting from grounds up to deploy their services and applications on InfiniBand based computing environments. At the end of the talk, it all looks very simple.. right ? But where to start.. that’s the fun part. Lets give some insight into these type of discussions.

Recall one of my previous blog where I focused on a few keywords – Consolidation, Isolation and Virtualization. This blog is build up on the discussion I presented there.

First things first !

Go through your server sizing activity and get an idea on how many applications you can fit into a compute machine based on its resources like CPU, memory, disk space etc. When you have a number of applications to deploy or even migrate from Ethernet environments, you would probably go with a model where you group some servers to perform certain set of functions. Right ? Ok, now you want to plan how these groups communicate with each other and at a lower level, how the servers within each group communicate with each other. So, all this work will create a basis for network design and architecture.

Network Design with InfiniBand

For simplicity, I will assume that we have total eight compute machines divided equally into four groups. And by now, you have already decided what applications run where. Networks can be classified into two main categories – Internal and External. As these terms suggest, internal networks are primarily used for communications like server management, patching, updates, synchronization, health checks etc and external networks are used to advertise and offer the running services to your outside world which may even be Internet.

Question – Will all eight machines be talking to the external network ? May be, may be not. Lets assume that we have a few select machines for this and they are categorized in group A. This group could be serving the purpose of a load balancer. Sure ! The two machines in this group may also be hosting some other service applications e.g. a name service and proxy. Sure ! Now, what I am trying to highlight here is that a machine may be deployed with multiple applications, those of which may have a need to participate in multiple networks.

So, the network participation is not driven by machine as a hardware.. but its a function of each running service or application inside the machine.

External Networks

Group A faces the external world and if this is a case of Exalogic hardware, the external access is made possible through Ethernet over InfiniBand or EoIB. If we have two different functions e.g. load balancer and a name service exposed to external world, we may wish to isolate them. This can be done by creating separate Virtual NICs and assigning their own IP addresses. These may even be on unique VLANs. If isolation is not needed, we can simply use same IP address. Most of the applications have a “Listen” feature which dictates what socket (IP and port) they bind to. Use of in-line firewalls on these external networks is not uncommon and this will provide security to our edge computing group A. We can even go a step ahead and write iptable rules on the machine’s inside group A for further security and hardening.

Internal Networks

Ok, so we took care of how the group A faces external world. In our example where we used load balancing as a function on the edge group A, it will have to communicate to internal groups where actual services may be running. And as I initially mentioned that we have already subdivided into three more groups – B, C and D. To build up a use case, lets assume that services to be load balanced are only deployed in group B. So that means, group A will be communicating to B on some specific sockets and we dont want any exposure from A to [C and D]. So, what do we do here ? We know that everything is connected over the same physical segment which is InfiniBand. Just like we will use VLAN in Ethernets, we will use Partition Keys here. The hosts in group A and B will share a common Partition key to allow a secured communication. Lets name this network as AB-10.

It may so happen that when machines in group B receives jobs to do from A, they need some help from machines in group C. Earlier, we excluded group C from network AB-10 for security. Hmm.. so what we do here is create a new network between group B and C just for this internal usage. Lets call it BC-20. And let me remind you that when you create a new network using VLAN or Partiton Keys, it means you have a new set of network interfaces and their associated IP addresses. This is what I discussed in my earlier blog – Networks and Virtualization.

What about group D ? Well, I left it alone on purpose just to demonstrate a use case that this group may be designated for some internal development and testing. But still, some network definitions must exist for this lone group D because there are two machines inside it and they may need to communicate to each other. Lets call it D-44. It will remain isloated from [A, B & C] as well as the external EoIB network created for A.

I have not forgotten about the NFS server.. so lets finish our conversation here. If all machines need to mount their respective shares from a common NFS server, then we can allow it to participate in all sub-networks AB-10, BC-20 and D-44. The access control policies can be enforced to allow only specific hosts to use the exported shares.

Wrap Up

What I have discussed here is that a set of machines can be grouped into functional roles which eventually drives the need of networks for communications – internal and external. Network isolation can be achieved in InfiniBand through Partition Keys just like VLANs in Ethernet. In-line firewalls can be used on EoIB network paths. Host level iptable rules can also be used if desired.

*About the author: Neeraj Gupta joined Oracle as part of Sun Microsystems acquisition, where he spent last 11 years specializing in InfiniBand, Ethernet, Security, HA and Telecom Computing Platforms. Prior to joining Sun, Neeraj spent 5 years in Telecom industry focusing on Internet Services and GSM Cellular Networks. Currently Neeraj is part of Oracle’s Engineered Systems team focusing on Networking and Maximum Availability Architecture.

Add Your Comment