OCI to Azure Interconnect with IPSEC backup

In some of my previous blogs, I presented a few advanced scenarios related to the OCI – Azure Interconnect service. If you missed them, here are some quick links:

OCI-Azure Interconnect advanced scenarios - part1

OCI-Azure Interconnect advanced scenarios - part2

In other blogs, we discussed OCI to Azure connectivity with the usage of IPSEC tunnels and what it takes to make it work.

OCI to Azure redundant IPSEC - part 1 - concepts

OCI to Azure redundant IPSEC - part 2 - demo

In this blog entry, we will combine the concepts and create an interesting use case: OCI – Azure Interconnect with IPSEC as a backup path.

Cloud connectivity services

As both the Interconnect and the IPSEC services are two different, standalone offerings, let’s see how each CSP interacts with them when both provide the same connectivity.
 

OCI
In OCI, the Interconnect, which uses the construct of the FastConnect, terminates logically in the Dynamic Routing Gateway. Similarly, any IPSEC connection will be attached to the DRG so the DRG, as the hub of all connectivity, will have to decide which path the flows will take. To understand this, we can review this document which talks about redundancy in OCI.

Inside that document we have this neat little table:

That table tells us that, for a given advertised subnet, the DRG will prefer the FastConnect service first, then move to IPSEC + BGP and look at IPSEC + Static routing last. For our scenario, that is perfect as it is the desired architecture: the DRG will always send the traffic over the Interconnect (FastConnect) and use IPSEC as a backup regardless of the IPSEC routing mechanism (BGP vs Static).

Azure

On the Azure side things are a little different:
- Azure uses different types of Virtual Network Gateways for each service, an ExpressRoute dedicated gateway for the Interconnect and an IPSEC dedicated gateway
- In order to achieve our desired architecture we will have to deploy both gateways in the VNET Gateway Subnet
While the connection point is a little different, Azure has a similar preference for the path, meaning if the same subnet is received in both gateways, Azure will always prefer to use the ER Gateway first. As there are a number of prerequisites to deploying this scenario in Azure, I will point out some of Microsoft’s documentation on the matter: Link1 and Link2.

The takeaway is that you need:
- A /27 or larger Gateway Subnet to accommodate both Gateways.
- The IPSEC gateway must be route based.
- The BGP ASN must be 65515 for the IPSEC connection.

That being said, it seems that both CSPs will prefer the Interconnect as a primary path and will use IPSEC as a backup without any extra configuration from us, which is great.

DEMO

For the demo part we will not go into the step-by-step procedure for deploying each service because that is extensively covered in the blogs above. In the end, the diagram should resemble this one:

Note that I will use a single IPSEC tunnel as backup (with static routing). Using the blogs mentioned above you can create multiple IPSEC tunnels to increase your redundancy level. You can deploy the Interconnect and IPSEC services in whichever order you like. In the end, you should have all connections UP.

Interconnect:

IPSEC:

Now that both the Interconnect and IPSEC are UP, let’s see how each CSP chooses the path.
For OCI, the easiest way is to go to the DRG Route Table of any attached VCN and click “Get all routes”.

As we can see, the Azure VNET subnet is available via both connections but the DRG has marked the IPSEC as “conflict” which means it’s used as a backup for the “Active” path.

For Azure, a good way to see the active path is to deploy a Virtual Machine, go in the network interface menu and choose the effective routes option.

While not immediately obvious, the gateways mentioned there are the Express Route Gateways (there are two for redundancy). We will see this when we test the path redundancy.

Let’s test the setup by disabling the Interconnect. We can do this easily in OCI by going to the Virtual Circuit menu and pressing “Deactivate” which will turn off BGP on the connection.

After a short time the Interconnect will show the BGP as down and everything will move over the IPSEC seamlessly.
On the DRG Route table of the VCN Attachment the routes changed to:

In Azure, the effective routes menu changed too, now clearly showing the flows going over the IPSEC VPN Gateway (as it mentiones the Public IP), different from the Gateway selected before.

And this concludes the demo!
As a final note, you can add as many IPSEC tunnels as you wish to act as backup for the Interconnect. The time to failover from Interconnect to IPSEC is dictated by the Interconnect BGP timers so it should not be more than 180 seconds.