Using Automatic Network Resource in Solaris Zone on Exalogic

This article will show you how to use Automatic Network (anet) resource to configure IPoIB and EoIB bonded interfaces in a Solaris zone on an Exalogic machine running Solaris 11.1.

Let’s start with a basic configuration of a zone with no network interface configured:

zonecfg:zoss01> info
zonename: zoss01
zonepath: /zones/zoss01
brand: solaris
autoboot: false
bootargs:
file-mac-profile:
pool:
limitpriv:
scheduling-class:
ip-type: exclusive
hostid:
fs-allowed:
rootzpool:
        storage: iscsi://192.168.10.15/luname.naa.600144F0B6FB373A0000531D230A0001

The name of the zone is “zoss01”,  it is a Solaris 11 native zone with brand set to “solaris” and it is running over iSCSI. This zone is configured with exclusive-ip that is the default value in Solaris 11.1, please refer to the documentation on the differences between exclusive-ip and shared-ip mode. Anet resource is available in  exclusive-ip mode only.

The following steps illustrate how to add IPoIB network interfaces to the zone using anet resource:

root@el01cn01:~# zonecfg -z zoss01
zonecfg:zoss01> add anet
zonecfg:zoss01:anet> set linkname=bond0_0
zonecfg:zoss01:anet> set lower-link=ibp0
zonecfg:zoss01:anet> set pkey=0xffff
zonecfg:zoss01:anet> end
zonecfg:zoss01> add anet
zonecfg:zoss01:anet> set linkname=bond0_1
zonecfg:zoss01:anet> set lower-link=ibp1
zonecfg:zoss01:anet> set pkey=0xffff
zonecfg:zoss01:anet> end
zonecfg:zoss01> exit

where linkname is the name of the interface that will be appeared in the zone, you may give it a different name; pkey is the partition key where 0xffff is the default partition key;  lower-link should be set to the corresponding physical link as indicated by the output of dladm command:

root@el01cn01:~# dladm
LINK                CLASS     MTU    STATE    OVER
igb0                phys      1500   up       --
igb2                phys      1500   unknown  --
igb1                phys      1500   unknown  --
igb3                phys      1500   unknown  --
usbecm0             phys      1500   up       --
ibp1                phys      65520  up       --
eoib0               phys      1500   up       --
eoib1               phys      1500   up       --
ibp0                phys      65520  up       --
bond0_0             part      65520  up       ibp0
bond0_1             part      65520  up       ibp1
vnic3066_0          vnic      1500   up       eoib0
vnic3066_1          vnic      1500   up       eoib1

IMPORTANT: please verify the zone configuration to make sure that the property “link-protection” is *not* specified, if it was set to any value (even if it was set to the default value “mac-nospoof”),  use clear command in zonecfg to unset this property.

The following steps illustrate how to add EoIB network interfaces to the zone using anet resource:

root@el01cn01:~# zonecfg -z zoss01
zonecfg:zoss01> add anet
zonecfg:zoss01:anet> set linkname=bond1_0
zonecfg:zoss01:anet> set lower-link=eoib0
zonecfg:zoss01:anet> set vlan-id=3066
zonecfg:zoss01:anet> end
zonecfg:zoss01> add anet
zonecfg:zoss01:anet> set linkname=bond1_1
zonecfg:zoss01:anet> set lower-link=eoib1
zonecfg:zoss01:anet> set vlan-id=3066
zonecfg:zoss01:anet> end
zonecfg:zoss01> exit

where linkname is the name of the interface that will be appeared in the zone, you may give it a different name; vlan-id is the vlan ID of the EoIB network, in this example, it is 3066, if there is no VLAN, this property is not required to set;  lower-link should be set to the corresponding physical link as indicated by the output of dladm command:

root@el01cn01:~# dladm
LINK                CLASS     MTU    STATE    OVER
igb0                phys      1500   up       --
igb2                phys      1500   unknown  --
igb1                phys      1500   unknown  --
igb3                phys      1500   unknown  --
usbecm0             phys      1500   up       --
ibp1                phys      65520  up       --
eoib0               phys      1500   up       --
eoib1               phys      1500   up       --
ibp0                phys      65520  up       --
bond0_0             part      65520  up       ibp0
bond0_1             part      65520  up       ibp1
vnic3066_0          vnic      1500   up       eoib0
vnic3066_1          vnic      1500   up       eoib1

When the zone is up and running, here is what the dladm command will return inside the zone:

root@zoss01:~# dladm
LINK                CLASS     MTU    STATE    OVER
bond0_0             part      65520  up       ?
bond0_1             part      65520  up       ?
bond1_0             vnic      1500   up       ?
bond1_1             vnic      1500   up       ?

The interfaces can now be used to configure bond0 and bond1.

Here is an example of how to configure bond0:

root@zoss01:~# ipadm create-ip bond0_0
root@zoss01:~# ipadm create-ip bond0_1
root@zoss01:~# ipadm create-ipmp bond0
root@zoss01:~# ipadm add-ipmp -i bond0_0 -i bond0_1 bond0
root@zoss01:~# ipadm set-ifprop -p standby=on -m ip bond0_1
root@zoss01:~# ipadm create-addr -T static -a 192.168.10.111/24 bond0/v4

Here is an example of how to configure bond1:

root@zoss01:~# ipadm create-ip bond1_0
root@zoss01:~# ipadm create-ip bond1_1
root@zoss01:~# ipadm create-ipmp bond1
root@zoss01:~# ipadm add-ipmp -i bond1_0 -i bond1_1 bond1
root@zoss01:~# ipadm set-ifprop -p standby=on -m ip bond1_1
root@zoss01:~# ipadm create-addr -T static -a 192.168.99.111/24 bond1/v4

Here is an example of what ipadm command will return after configuration:

root@zoss01:~# ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
bond0             ipmp       ok           --         --
   bond0/v4       static     ok           --         192.168.10.111/24
bond1             ipmp       ok           --         --
   bond1/v4       static     ok           --         192.168.99.111/24
bond0_0           ip         ok           bond0      --
bond0_1           ip         ok           bond0      --
bond1_0           ip         ok           bond1      --
bond1_1           ip         ok           bond1      --
lo0               loopback   ok           --         --
   lo0/v4         static     ok           --         127.0.0.1/8
   lo0/v6         static     ok           --         ::1/128

By using anet resource, we can use the same link-name for different zones, in our example above, the underlying interfaces for bond0 are called bond0_0 and bond0_1, the same name can be used across different zones, system administrator may find it easier to manage and simpler to automate administrative tasks.

From the global zone point of view, zone-specific network interfaces will show up only when a zone is up and running, this can be illustrated by running dladm command in the global zone:

root@el01cn01:~# dladm
LINK                CLASS     MTU    STATE    OVER
igb0                phys      1500   up       --
igb2                phys      1500   unknown  --
igb1                phys      1500   unknown  --
igb3                phys      1500   unknown  --
usbecm0             phys      1500   up       --
ibp1                phys      65520  up       --
eoib0               phys      1500   up       --
eoib1               phys      1500   up       --
ibp0                phys      65520  up       --
bond0_0             part      65520  up       ibp0
bond0_1             part      65520  up       ibp1
vnic3066_0          vnic      1500   up       eoib0
vnic3066_1          vnic      1500   up       eoib1
zoss01/bond0_0      part      65520  up       ibp0
zoss01/bond0_1      part      65520  up       ibp1
zoss01/bond1_0      vnic      1500   up       eoib0
zoss01/bond1_1      vnic      1500   up       eoib1

The last four rows will only appear when zoss01 is running and they clearly indicate which zone they belong to. Without using anet resources, network interfaces have to be created explicitly in the global zone and they will appear in the command output no matter the zone is running or not.

 

Add Your Comment