IPSEC over OCI to Azure Interconnect

In this blog entry, we will talk about encrypting flows on the private connection between the Oracle Cloud and Microsoft Azure (called Interconnect) with IPSEC. Also, please check my previous blogs related to connectivity between OCI and Azure:
OCI to Azure Interconnect advanced scenarios
OCI to Azure redundant IPSEC
OCI to Azure Interconnect with IPSEC backup

Private IPSEC

IPSEC is mostly used to encrypt traffic that flows over the public Internet. However, with the large scale adoption of hybrid-cloud and multi-cloud architectures, there are a lot of point-to-point, private, dedicated connections in use. While each CSP calls them differently (FastConnect, ExpressRoute, DirectConnect), they all do the same thing and are mostly unencrypted (some scenarios do support MACSEC encryption). Even though most of the flows running through those links are TLS/SSL encrypted, you can find a case where you want network layer encryption. In those cases we can use IPSEC over the private links. Let’s first understand how each party does this.

Azure

In Azure, things are pretty straightforward:
- The VPN Gateway that is used for Internet IPSEC, thus it has a Public IP, can also be configured to have a private IP from the Gateway subnet inside the VNET.
- While you deploy the IPSEC connection, there is a setting to tell it to use the private Gateway IP;
- If the private IP used to declare the Local Network Gateway (customer end of the IPSEC) is reachable via the ExpressRoute link then IPSEC should establish.

OCI

In OCI, this service was recently released and you can read this excellent blog on it. The implementation highlights are:
- When you declare the Customer-premises equipment (customer end of the IPSEC) you have a checkbox to say this will be used for IPSEC over FastConnect;
- When you create the IPSEC tunnel you are asked which FastConnect link it will use and you are also asked for a Dynamic Routing Gateway Loopback IP which will be the IPSEC peer IP;
- The moment you assign the DRG Loopback IP, it will start to be advertised over the specified FastConnect link so the other end gets a route toward it;
- The only restriction is that the VPN and the FastConnect it uses cannot use the same DRG Route table and Import Distribution lists so you need to create new ones, dedicated to the VPN;

OCI – Azure Private IPSEC

These IPSEC functions were designed for connectivity to datacenters where it is implied we have full control of the CPEs, mainly related to routing advertisements. Trying to build IPSEC over the Interconnect service does present some challenges, mainly for routing. Let’s see why.
When you build the Interconnect, which leverages the FastConnect construct in OCI and ExpressRoute in Azure, the following will happen if no extra configuration is done:
- The OCI DRG will advertise over BGP all the connected VCNs CIDRs or Subnets;
- The Express Route Gateway in Azure will advertise the VNET CIDR and, optionally, all the connected spoke VNETs CIDRs;
Now, when we build the IPSEC tunnel, if we use BGP we will have no control over what is advertised so the same routes advertised over Interconnect will also be advertised over the IPSEC. In this case, both CSPs will prefer the FastConnect/Expressroute constructs and not the IPSEC, hence no encryption will happen. The way around it is to use route-based IPSEC with STATIC ROUTES, where the static routes have a longer prefix (hence better match) and will be preferred. While BGP should always be preferred, the cloud constructs that we use to build this prevent us from using it both on the Interconnect and the IPSEC which runs on top of it. This can be used to our advantage as we can choose which subnets will be sent through the IPSEC and which will be sent directly over the Interconnect.

As a summary, here is what we need to do:
1. Build Interconnect which uses BGP that advertises OCI VCN and Azure VNET full CIDRs. Note that, by default, the DRG will advertise the individual subnets of the VCN and that there is a specific setting in the DRG attachment that changes that to advertising the full VCN CIDR.
2. Build IPSEC over Interconnect using static routing in which we define manually which subnets are sent through the IPSEC.

DEMO

Let’s put everything we discussed on a diagram and implement it:

The main steps to implement this will be detailed below. Be aware that I will skim over some steps as some knowledge of both OCI and Azure is expected.

The Interconnect

Before diving into IPSEC, you need to have the Interconnect up between OCI and Azure. You can use this guide to deploy it. For this demo, I have the following setup:
a. One VNET in Azure with the CIDR 192.168.0.0/24, one APP Subnet with the CIDR 192.168.0.0/27 and one Gateway Subnet with 192.168.0.32/27.
b. One ExpressRoute Gateway in the Gateway Subnet linked with the Express route circuit.
c. One Test VM in the APP Subnet.
d. In OCI, I have a VCN with the CIDR 10.0.0.0/24 and a subnet with 10.0.0.0/26. The Subnet also contains a Test Compute.
e. The VCN is attached to a DRG which also has the FastConnect/Interconnect attached.

For this setup to work, we need the OCI DRG to advertise the full VCN CIDR over the Interconnect. This is a setting in the VCN DRG Attachment:

Setting up IPSEC

1. The Azure VPN Gateway
The first step is to deploy a VPN Gateway on the Azure side. According to Microsoft’s documentation, only certain Gateway SKUs support this feature: VpnGw1AZ, VpnGw2AZ, VpnGw3AZ, VpnGw4AZ, VpnGw5AZ. Deploy a Gateway using one of those SKUs.

Wait for the deployment to complete and go to the resources Configuration menu and Enable the Gateway’s Private IP functionality.

After the Gateway gets updated, go to the Properties tab and take a note on the assigned Private IP.

In this case, the Gateway was assigned 192.168.0.38. Next stop: the OCI Console.

2. The OCI Customer-premises equipment
The target of the VPN, for OCI, will be the Azure VPN Gateway Private IP. Let’s go to Networking -> Customer Connectivity -> Customer-premises equipment and create a new CPE, as in the PIC below:

Make sure you check “Allow IPSec over FastConnect”

3. The DRG route table and Import Route Distribution
By default, both the FastConnect and the IPSEC tunnels will use the same DRG route table. In the case of IPSEC over FastConnect that is not possible so we need to create a new DRG route table that uses a new Import Route Distribution specifically for this service. Go to the DRG that has the Interconnect and create a new Import Route distribution:

Next, create a new DRG route table that uses that Import Route distribution:

4. The OCI VPN
We are now ready to deploy the OCI VPN service which will consist of two IPSEC tunnels to the Azure Gateway. Go to the Networking -> Customer Connectivity -> Site-to-Site VPN menu and create a new IPSEC connection. Details for the connection are:
CPE – the one we created earlier
DRG – the DRG that holds the Interconnect
Routes to your on-premises network: the subnets inside the Azure VNET that you want to use the IPSEC service. In my case, this is the APP Subnet (192.168.0.0/27).

Next, in the same page, we have the tunnel configuration.
The Oracle tunnel headend IP is the actual DRG Loopback interface which will be used for the IPSEC peering. Use an IP there that is not in use anywhere else because the DRG will advertise it as a /32 which will make it more likely to be selected. Make sure you select the correct Virtual Circuit (the Interconnect) and that you use the DRG route table created before.

Next, Select advanced options and input custom IPSEC parameters as detailed in this blog.

Repeat the procedure above and configure tunnel 2.

After you press create, you should see the following:

5. Azure Local Network Gateways
Next, we move back to the Azure Console to define the Local Network Gateways (OCI endpoints) that we will use in the IPSEC creation. Note that you need to define the OCI subnet which will be included in the routing.

6. Azure IPSEC Connections.
Next, go to the VPN Gateway, click Connections and +Add to add a new IPSEC connection.

In the settings tab, make sure you check “Use Azure Private IP Address”. Also, make sure you match the IPSEC parameters you defined on the OCI side.

Repeat the procedure for tunnel 2 (make sure you select the other Local Network Gateway).
After you deploy both tunnels, you should see the following status, in Azure:

In OCI, we can see:

7. Equal Cost multi-path (optional)
As we created 2 tunnels between the CSPs, both with the same configuration, we now have 2 paths for the same routes. The Azure VPN Gateway does ECMP by default but on the OCI side you must enable it. You do this on the VCN Attachment DRG Route table:

Unless you have a good reason not to enable it, this will roughly double the available bandwidth if you use multiple sources/destinations.

With this, the deployment is now complete, we have created IPSEC tunnels over the Interconnect. Let’s check how the routing looks.
On the Azure side, we can go on the network interface of the Virtual Machine and see effective routes.

As we can see, the OCI VCN CIDR (10.0.0.0/24) and the DRG Loopback interfaces are being advertised by the ExpressRoute (Interconnect) but the subnet IP space (10.0.0.0/26) is being advertised by the VPN/IPSEC Gateway.

On the OCI side, the best place to look is the DRG VCN attachment route table. Let’s do a “get all route rules”.

We can clearly see the /24 Azure VNET over the Virtual Circuit and the smaller /27 subnet going over the IPSEC tunnels.

Performance

The performance (throughput) of this connection will be determined by multiple factors:
- The Azure side Gateway SKUs – both the ExpressRoute and the VPN Gateways can be deployed with multiple SKUs which directly affects performance. Check the Microsoft documentation to select the one that meets your needs.
- The Circuit bandwidth – it is clear that the overall throughput of the IPSEC connections cannot go above the allocated bandwidth of the Interconnect circuit.
- ECMP enabled – when we enable ECMP on the OCI side we aggregate the throughput of all tunnels, provided we use multiple streams
- Single stream performance – this one is the hardest to quantify as it is directly linked to the protocol used, the latency of the link, the MTU, and so on.

For the demo scenario, both OCI and Azure regions are in Frankfurt. The latency between the test hosts is 2-3 ms. Let’s see what a single stream IPERF3 test shows:

As we can see, a single stream reaches ~450 Mbps. As we have 2 tunnels and the underlying circuit is configured for 1Gbps bandwidth, let’s test with 4 parallel streams.

The multiple streams test, which mimics real-world deployments better, shows us we can go up to 900Mbps if we have 2 IPSEC tunnels and ECMP.
Again, take special care when selecting the Azure SKUs and be aware that high latency will have a big impact on throughput.

And this concludes the demo!