X

Best Practices from Oracle Development's A‑Team

OCI Layer 2 Networking Support

Javier Ramirez
Principal Cloud Solution Architect

The purpose of this blog is to showcase a new feature introduced by OCI using which you can deploy the OCI Layer 2 Networking (VLANs) within your Virtual Cloud Network (VCN).

Disclaimer: This feature is not generally available yet, if you have a use case please work with your technical team to get your tenancy whitelisted to use this feature.

 

Disclaimer: This blog showcase OCI Layer 2 Networking Support and not the specific vendor use in the example. The customer needs to work with the specific vendor to make sure its product will work with OCI Layer2 Networking and will be supported by the vendor.

 

The OCI Layer 2 Networking feature within your VCN can be used to create an HA cluster where the application/software manages the state of the cluster. With the OCI Layer 2 Networking feature, you can deploy an HA solution in the same way you deploy it on-prem. If you deploy a pair of Firewalls (FWs) in a HA configuration today your FWs must interact with OCI via the API to move the Virtual IP (VIP) from the active to the standby FW when a failure occurs, this is accomplished using Instance Principles (Dynamic Groups and Policies).  Different vendors have taken different approaches to manage such deployments and some vendors have implemented such API calls in their code while some other vendors do not support HA configurations in OCI.

To take advantage of this new feature, you need to make some configuration changes via the Oracle Console to be able to interact with the rest of your VCN or external networks for example the Internet or on-prem. Keep in mind that when using OCI Layer 2 Networking the failover control process is managed by the virtual devices you are deploying using protocols such as HSRP, VRRP, or multicast.

 

Use Case

There are several use cases that can benefit from the OCI Layer 2 Networking feature. For the purpose of this blog, I’m going to use two Cisco ASAv firewalls in HA mode (active/passive) and show you how the new Layer 2 network can be used to set up the HA cluster and how it allows the devices to failover gracefully similar to on-prem. The diagram below shows the setup used in this exercise.

On the diagram above dashed lines represent standard subnets (yellow and green) while the solid lines are the OCI Layer 2 Networks (VLANs) (blue, red, black).

 

VLANs and Subnets

The first step is to create your subnets and VLANs. In the Oracle Console, under your VCN you need to create the subnets (private or public); that is where you will also create the VLANs with their corresponding VLAN tag and Gateway CIDR as per your requirements. A VLAN also needs a routing table. You can define a new one or just use the default table. You can always go back and revise this decision. Regarding security, VLANs only support Network Security Groups (NSGs), they do not support Security Lists. An NSG can be assigned at a later time.

VLANs in the Hub VCN

VLANs in the spoke VCN

Disclaimer: At the time of writing this blog, OCI does NOT support instantiating a Virtual Machine (VM) with its primary interface (vNIC) on an OCI Layer 2 Network. Typically the first interface becomes the management interface for the virtual appliance. Be aware of this when deploying your HA pair.

 

Next, you need to instantiate the Cisco ASAs. The primary interface (vNIC) can’t be placed on a VLAN at this time, you can only use a subnet that is why the Management interface for the Cisco ASAs is part of the MGMT subnet (green) in the diagram above. As I will show later, this would lead to an interesting issue.

After your Cisco ASAs are up, add the additional vNICs to each Cisco ASA and deploy those vNICs to the corresponding VLANs. Per diagram above, add 3 vNICs (Outside (blue), HA (red), and Inside(black)). Make sure that when you build the additional vNICs on each Cisco ASA, this is done in the same order. This process applies to any vendor where you need to deploy multiple vNICs for an HA pair.

Once the vNICs are set up, reboot the VM and log into your Cisco ASA primary device and configure the interfaces. Below is the interface configuration for the VMs in the Oracle Console.

Primary FW

Secondary FW

Below is the interface configuration on the Cisco ASA.

interface GigabitEthernet0/0
 nameif Outside
 security-level 0
 ip address 10.0.100.50 255.255.255.240 standby 10.0.100.51 
!
interface GigabitEthernet0/1
 description LAN/STATE Failover Interface
!
interface GigabitEthernet0/2
 nameif Inside
 security-level 100
 ip address 10.0.101.18 255.255.255.240 standby 10.0.101.19 
!
interface Management0/0
 no management-only
 nameif management
 security-level 100
 ip address 10.0.100.22 255.255.255.240 standby 10.0.100.25

Once failover is set up on both Cisco ASAs, the configuration from the primary (active) will be replicated to the secondary (standby) Cisco ASA.

Note: As shown above the management interface is also configured for failover. When failover occurs Cisco also moves the management IPs. This causes an issue as the management interface (vNIC) is deployed on a subnet and not on a VLAN. The IP addresses in a subnet are owned by OCI and can’t be moved unless the Cisco ASA via an API call tells OCI to move them. Other vendors do not move the management IPs when a failover occurs.

 

The pictures below show how the Cisco ASA failover configuration looks like after the HA configuration is done. When the primary is active, and the secondary is in standby mode. Each management interface (10.0.100.22 and 10.0.100.25) has its own IP address as originally assigned by OCI and you can SSH or ping them.

Primary (Active) FW

ASA1# show int ip br
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         10.0.100.50     YES manual up                    up  
GigabitEthernet0/1         10.0.100.35     YES unset  up                    up  
GigabitEthernet0/2         10.0.101.18     YES manual up                    up  
Internal-Data0/0             169.254.1.1     YES unset  up                    up  
Management0/0                   10.0.100.22     YES manual up                    up  
ASA1# 

Secondary (Standby) FW

ASA1# show int ip br
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         10.0.100.51     YES manual up                    up  
GigabitEthernet0/1         10.0.100.34     YES unset  up                    up  
GigabitEthernet0/2         10.0.101.19     YES manual up                    up  
Internal-Data0/0             169.254.1.1     YES unset  up                    up  
Management0/0                   10.0.100.25     YES manual up                    up  
ASA1#

Now if I force the failover process, all the IP addresses for all the interfaces will switch over which is fine from the interface point of view as you can see in the next two pictures.

Primary (Active) FW

ASA1# show int ip br
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         10.0.100.51     YES manual up                    up  
GigabitEthernet0/1         10.0.100.35     YES unset  up                    up  
GigabitEthernet0/2         10.0.101.19     YES manual up                    up  
Internal-Data0/0             169.254.1.1     YES unset  up                    up  
Management0/0                   10.0.100.25     YES manual up                    up  
ASA1#

Secondary (Standby) FW

ASA1# show int ip br
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         10.0.100.50     YES manual up                    up  
GigabitEthernet0/1         10.0.100.34     YES unset  up                    up  
GigabitEthernet0/2         10.0.101.18     YES manual up                    up  
Internal-Data0/0             169.254.1.1     YES unset  up                    up  
Management0/0                   10.0.100.22     YES manual up                    up  
ASA1#

But when you check the failover state of the Cisco ASAs (see next two pictures), you could clearly see an issue. The management interface is stuck in the Waiting status and will never transition to the Monitored mode. The reason for this is because as mentioned earlier, OCI owns the IPs within a subnet. In this case, the Cisco ASAs moved the IPs between the devices, but OCI has not moved them. From OCI's point of view, 10.0.100.22 is still assigned to the primary FW and 10.0.100.25 is assigned to the secondary FW. For this reason, the management interface will become useless only when the secondary is active. If the primary is active then this issue goes away as the IPs return to the original configuration. This issue will affect ONLY the management of the Cisco ASAs and does not affect the functionality of the Cisco ASAs. A workaround is to manage the Cisco ASA via the console port or set up other interfaces of the firewall for management.

Primary (Active) FW

ASA1# show failover
Failover On 
Failover unit Primary
Failover LAN Interface: HA GigabitEthernet0/1 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 3 of 1285 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.15(1)1, Mate 9.15(1)1
Serial Number: Ours 9AA2C7TE3P0, Mate 9AFS6EM0RV0
Last Failover at: 14:47:32 UTC May 14 2021
	This host: Primary - Standby Ready 
		Active time: 649 (sec)
		slot 0: ASAv hw/sw rev (/9.15(1)1) status (Up Sys)
		  Interface Outside (10.0.100.51): Normal (Monitored)
		  Interface management (10.0.100.25): Unknown (Waiting)
		  Interface Inside (10.0.101.19): Normal (Monitored)
	Other host: Secondary - Active 
		Active time: 198 (sec)
		  Interface Outside (10.0.100.50): Normal (Monitored)
		  Interface management (10.0.100.22): Unknown (Waiting)
		  Interface Inside (10.0.101.18): Normal (Monitored)

Secondary (Standby) FW

ASA1# show failover
ASA1# show failover
Failover On 
Failover unit Secondary
Failover LAN Interface: HA GigabitEthernet0/1 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 3 of 1285 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.15(1)1, Mate 9.15(1)1
Serial Number: Ours 9AFS6EM0RV0, Mate 9AA2C7TE3P0
Last Failover at: 14:47:28 UTC May 14 2021
	This host: Secondary - Active 
		Active time: 216 (sec)
		slot 0: ASAv hw/sw rev (/9.15(1)1) status (Up Sys)
		  Interface Outside (10.0.100.50): Normal (Monitored)
		  Interface Inside (10.0.101.18): Normal (Monitored)
		  Interface management (10.0.100.22): Unknown (Waiting)
	Other host: Primary - Standby Ready 
		Active time: 649 (sec)
		  Interface Outside (10.0.100.51): Normal (Monitored)
		  Interface Inside (10.0.101.19): Normal (Monitored)
		  Interface management (10.0.100.25): Unknown (Waiting)
Note: There are other use cases where this might not be an issue, for example, if you have two virtual routers using HSRP or VRRP, the failover will happen between the interfaces, but the management interface typically does not participate in HSRP or VRRP process and everything should work just fine. Make sure the primary interface (vNIC) on the network appliance does not participate in the failover process. It is recommended that you check with your vendor and fully understand how the failover affects the management or primary interface (vNIC) in the instance. Some vendors do not failover the management interface so this will not be an issue in those cases.

 

Up to this point, the failover process is active between the Cisco ASAs, but you still need to allow traffic to pass through the firewall, so the Inside and Outside interfaces need to be able to route and interact with the rest of your VCN, on-prem, and perhaps the Internet.

 

External Connectivity

When your requirement is to interact with something external to the VLAN, you need to define External Access for each VLAN. This is accomplished via the Oracle Console. For this example, on the Outside VLAN, I defined the following external access.

As shown above, I defined the active and standby IP addresses for the Outside interface. I also assigned a public IP address to the VIP in case I need to allow traffic from the Internet or build an IPSec tunnel to the outside world.

For the HA VLAN, I did not define any External Access because the HA VLAN is only used for communication between the two Cisco ASAs. For the Inside VLAN, I defined similar external access as what I defined for the Outside VLAN with the exception of the public IP address assigned to the VIP.

Now that you have defined your external access you need to make sure that the route table associate with each VLAN provides proper routing to external endpoints.

For the Outside and HA VLANs, I am going to use the default route table on the hub VCN. The HA VLAN does not need any external communications, so it does not matter which route table is assigned to it. The route table below is assigned to the Outside VLAN, take a look at row 5 and see how I route the traffic to the spoke VCN by pointing the route to the VIP of the Outside interface.

The Inside VLAN is also set with the default route table of the spoke VCN and the second row shows the route pointing to the VIP of the Inside interface to get back to the hub VCN.

From a security perspective also make sure you create/assign an NSG with the proper controls to allow the traffic to the VLAN. I created the following NSGs:

For the HA VLAN, you can restrict access only to the specific ports absolutely required by the virtual devices to establish and maintain the HA relationship, in this case for demonstration only I left it wide open which is not recommended in a real-world network.

For the Outside VLAN

For the Inside VLAN

 

Tests

Here is a simple test to demonstrate the HA functionality after deploying the new feature.

1) Ping from dr-vm2 (10.0.101.7) on the spoke VCN (Inside) to vmdr (10.0.100.2) on the Hub VCN (Outside) through the Cisco ASA. The ping is successful

[opc@dr-vm2 ~]$ ping 10.0.100.2
PING 10.0.100.2 (10.0.100.2) 56(84) bytes of data.
64 bytes from 10.0.100.2: icmp_seq=1 ttl=64 time=1.25 ms
64 bytes from 10.0.100.2: icmp_seq=2 ttl=64 time=1.48 ms
64 bytes from 10.0.100.2: icmp_seq=3 ttl=64 time=1.12 ms
64 bytes from 10.0.100.2: icmp_seq=4 ttl=64 time=1.39 ms
64 bytes from 10.0.100.2: icmp_seq=5 ttl=64 time=1.09 ms
^C
--- 10.0.100.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 1.094/1.269/1.484/0.156 ms
[opc@dr-vm2 ~]$

2) Maintain the ping in test 1 and force failover from the standby Cisco ASA to take over. Zero packets are lost during the transition

71 packets transmitted, 71 received, 0% packet lost, time 70411ms

3) Maintain the ping in test 1 and force the failover again by rebooting the active Cisco ASA (secondary)

78 packets transmitted, 64 received, 14 packet lost, time 77433ms

4) Ping from vmdr (10.0.100.2) on the Hub VCN (Outside) through the Cisco ASA to dr-vm2 (10.0.101.7) on the spoke VCN (Inside). The test fails because the security policy on the Cisco ASA is not allowing this traffic. The log shows the traffic is denied.

[opc@vmdr ~]$ ping 10.0.101.7
PING 10.0.101.7 (10.0.101.7) 56(84) bytes of data.
^C
--- 10.0.101.7 ping statistics ---
20 packets transmitted, 0 received, 100% packet loss, time 19491ms

[opc@vmdr ~]$

%ASA-3-106014: Deny inbound icmp src Outside:10.0.100.2 dst Inside:10.0.101.7 (type 8, code 0)
%ASA-3-106014: Deny inbound icmp src Outside:10.0.100.2 dst Inside:10.0.101.7 (type 8, code 0)

 

Conclusion

The OCI Layer 2 Networking feature can be very useful in certain use cases. For example, it can be a powerful tool to set up HA pairs within your VCN to support network, security, and SD-WAN devices. As mentioned above if you have a use case where you need to use OCI Layer 2 Networking on your tenancy, talk to your Oracle team and your vendor. We can review your requirements, make sure the solution is doable and whitelist your tenancy to use OCI Layer 2 Networking.

 

Check other blogs from Javier Ramirez

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha