Solaris on Exalogic – Reverse the active/pasive interface of an IPMP group

In a customer engagement, I found that that their bond0 and bond1 configurations look like this:

root@el01cn01:~# ipmpstat -i
INTERFACE   ACTIVE  GROUP       FLAGS     LINK      PROBE     STATE
eoib0       no      bond1       is—–   up        disabled  ok
eoib1       yes     bond1       –mb—   up        disabled  ok
bond0_0     yes     bond0       –mb—   up        disabled  ok
bond0_1     no      bond0       is—–   up        disabled  ok

notice that the active interface for bond1 is eoib1 while the active interface for bond0 is bond0_0

Although it is perfectly fine for the EoIB traffic and IPoIB traffic to go over different IB gateway, customer want like to re-configure it so that both type of traffic will go through the same IB gateway.

First of all, we turn on “standby” property for eoib1 and turn off “standby” property for eoib0 with the following commands:

root@el01cn01:~# ipadm set-ifprop -p standby=on -m ip eoib1
root@el01cn01:~# ipadm set-ifprop -p standby=off -m ip eoib0

The status of the IPMP groups look like this now:

root@el01cn01:~# ipmpstat -i
INTERFACE   ACTIVE  GROUP       FLAGS     LINK      PROBE     STATE
eoib0       yes     bond1       ——-   up        disabled  ok
eoib1       yes     bond1       -smb—   up        disabled  ok
bond0_0     yes     bond0       –mb—   up        disabled  ok
bond0_1     no      bond0       is—–   up        disabled  ok

Then we forced a failover by detaching eoib1 from bond1 with the following command:

root@el01cn01:~# if_mpadm -d eoib1

The status of the IPMP groups look like this now:

root@el01cn01:~# ipmpstat -i
INTERFACE   ACTIVE  GROUP       FLAGS     LINK      PROBE     STATE
eoib0       yes     bond1       –mb—   up        disabled  ok
eoib1       no      bond1       -s—d-   up        disabled  offline
bond0_0     yes     bond0       –mb—   up        disabled  ok
bond0_1     no      bond0       is—–   up        disabled  ok

Then we re-attach eoib1 back to bond1, because it is a standby interface, failback will not happen:

root@el01cn01:~# if_mpadm -r eoib1

That’s how it looks like after the re-configuration:

root@el01cn01:~# ipmpstat -i
INTERFACE   ACTIVE  GROUP       FLAGS     LINK      PROBE     STATE
eoib0       yes     bond1       –mb—   up        disabled  ok
eoib1       no      bond1       is—–   up        disabled  ok
bond0_0     yes     bond0       –mb—   up        disabled  ok
bond0_1     no      bond0       is—–   up        disabled  ok

Exalogic Virtual Tea Break Snippets – Allocating Static IP Addresses

By default once a Network has been created within the Enterprise
Manager Ops Centre (EMOC) it can be allocated to vServers during
their creation. At this point an IP Address will be allocated
automatically from the pool of Allocated IPs associated with the
Network and Account combination.

In many customer solutions the vServers will need to be allocated
a specific IP address so that they can be accessed externally at a
know location. To achieve this we must Allocate a number of vIPs
within the range allocated to the Account. This is done on an
Account by Account basis as follows.

Exalogic Virtual Tea Break Snippets – Creating Networks

In the majority or Real World scenarios we will need to access
the Virtual Servers running within the Exalogic from an external
client network. To facilitate this we will want to leverage the
10Gb Ethernet connection and hence we will need to create 1 or
more EoIB networks that can be accessed by the Virtual Servers.

During the installation of the Exalogic 2.0.1 Virtual environment
we create a single “EoIB-external-mgmt” network that we could, in
theory, use to access the Virtual Servers we create. Although this
is possible, assuming it has enough IP Address, this would be bad
practice because this network is intended solely for management
functionality and access to the Control VMs. Therefore to provide
the Virtual Servers with external Ethernet access we will need to
create additional EoIB interfaces. Each of these will need to be
VLAN tagged to provide network isolation and partitioning.

InfiniBand Enabled Diskless PXE Boot

If you ever need to bring up a computer with InfiniBand networking
capabilities and diagnostic tools, without even going through any
installation on its hard disk, then please read on. In this article, I
am going to talk about how to boot a computer…

Deployments over InfiniBand Infrastructure

What actually drives network requirements ? Use of InfiniBand Partition
Keys to create isolation between functional groups of servers. Sample
use case illustrating how to design internal InfiniBand networks.
Application consolidations and their asso…

Bonding Parameters Based on Network Layout

Technical discussion on how to adjust Linux bonding configuration
parameters depending on how a host participates in a given network.
Limitations of link level failure detections and an easy solution within
bonding driver to overcome it.
[Read Mo…

Networks and Virtualization

Technical discussion on an approach to achieve virtualization through
network layers. Comparison between InfiniBand Partitions and Ethernet
VLANs. Consolidation and Isolation of logical network channels leading
to virtualization. How to satisfy com…

InfiniBand Building Blocks

In the context of Oracle’s Engineered Systems, I am going to explain how
an InfiniBand network is actually created using various building blocks
of hardware and software. Includes a very basic view of a host
participating in IB network and their con…

InfiniBand: What about it ?

Neeraj Gupta from our Networking team introduces Infiniband.
[Read More]