IPSec over FastConnect, a deep dive approach

I need to recognize that I counted the days for this feature to be launched and for me to write a blog about IPSec over private FastConnect, one of the features we recently launched to complete the security suite running on top of FastConnect Private Virtual Circuits.

Welcome to the IPSec over FastConnect, a deep dive approach for a great security feature OCI launched.

Even if the FastConnect private peering is not running over the Internet being a dedicate private line between OCI and the customer on-premises, there are still requirements for the traffic passing through the private FastConnect virtual circuit to be encrypted. Some time ago we launched MACsec for Direct FastConnect (cross-connects) but the MACsec was not available for Partner private virtual circuits. That being said, the IPSec over private FastConnect filled the gap and completes the security suite for FastConnect.

The scope of this blog is not specifically to explain how the feature is being configured, but to analyze different new objects added, how the signaling is done and how the encrypted/non-encrypted traffic is flowing, mainly, all the new information introduced with this security feature. The configuration part is listed under Configuring IPSec over FastConnect section.

All the magic is implemented using the upgraded DRG, so, from the very beginning we need to say that the feature will work only on the upgraded DRG.

A private FastConnect virtual circuit can support two types of configuration named: All Traffic and IPSec over FastConnect traffic only. In the next section we will analyze both and we will see the difference between them.

1. IPSec over FastConnect traffic only - TransportMode

The first FastConnect virtual circuit option we will analyze is IPSec over FastConnect traffic only. This mode is called TransportMode because the virtual circuit will be used only to transport the encrypted traffic. Now, let's talk about the IPSec encrypted traffic.

How it looks?

The red squared source and destination IP addresses are represented by the IPSec tunnel headend and we are calling it outer IP header, or the IP header exposed to routers performing the route lookup.  The original IP header is encrypted, it is not visible, thus the TransportMode name. The FastConnect will be used only to transport the encrypted packets and the IP forwarding will be based on the tunnel headend IP addresses of the IPSec peers until the packet reaches one end of the peers to perform traffic decryption.

Non-encrypted traffic cannot be sent over the virtual circuit in this mode. Some restrictions will be automatically enforced by OCI for the non-encrypted traffic to be dropped (we will analyze the restrictions later).

Let's go ahead and analyze some technical details based on the simple but interesting networking topology:

The Provider or Direct Cross-Connect FastConnect private virtual circuit is configured (in this example Provider Virtual Circuit is used) with IPSec over FastConnect traffic only configuration:

The IPSec tunnel running on the private virtual circuit is configured between OCI VPNaaS endpoint at 10.10.0.1 and the customer on-premises firewall at 172.16.0.10. The OCI VM at 192.168.1.3 should be able to send encrypted traffic to on-premises user at 172.16.0.27 (and of course vice-versa).

Note: If you plan to use the On-premises FastConnect BGP IP address also for the IPSec tunnel IP address, please make sure you are advertising it to OCI either as a /32 IP address or /30 subnet. OCI by default does not redistribute the FastConnect /30 BGP peering subnet to any of the DRG route table attachments.

From our previous blogs related to DRG, we are familiar with some of the DRG attachments like VCN attachment, FC VC attachment and IPSec attachment. These are well-known type of attachments, broadly discussed and analyzed in the past. However in the networking diagram listed above there is a brand new type of attachment, introduced for IPSec over FastConnect and that is called the Loopback attachment. If we remember from the traditional networking, the "loopback interfaces" helped us to resolve many problems and also to have consistent routing configurations.

As we can observe, there are two pieces that needs to work together, the first one is the FC VC and the second is the IPSec tunnel on the IPSec service. There should be a way to make the two pieces work together -> the Loopback attachment.

The Loopback attachment was designed and introduced to permit:

    1. The encrypted traffic to flow between the FC VC and the IPSec tunnel and back to the DRG after performing encryption/decryption;

    2. To provide to the DRG the IPSec tunnels IP addresses for establishing the IPSec tunnels with the IPSec peer (OCI tunnel head ends);

The routing using the Loopback attachment is managed by OCI and cannot be changed by tenancy admins.

With this new approach, OCI has implemented a rule for avoiding the routing loops:

    1. The IPSec attachments and FastConnect VC attachments must use separate route tables with separate import route-distribution;

    2. The route table's of the private IPSec tunnel attachment and the private FastConnect VC attachment cannot have import route-distribution statements to import routes from each other;

Let's see it in action.

Encrypted traffic from OCI VM at 192.168.1.3 to on-premises host at 172.16.0.27

The above networking diagram is analyzing how the traffic is flowing from the OCI VM at 192.168.1.3 to 172.16.0.27. More than that, below we will have the content of each attachment route table used for a better understanding of the process.

Once the traffic is generated by 192.168.1.3 will arrive at the DRG VCN attachment route table:

The IPSec tunnel attachment is used as a next-hop attachment for 172.16.0.16/28.

Based on the above route rule, the unencrypted traffic is sent to the OCI IPSec service defined using the VPNaaS configuration process at 10.10.0.1. Here, the traffic is encrypted resulting an encrypted IP packet where the visible source and destination IP addresses are 10.10.0.1 and 172.16.0.10 (representing the IPSec tunnel head ends). Let's continue the logic.

The IPSec service is delivering the encrypted packet to the Loopback attachment. How the Loopback attachment route table looks like? Let's find out:

Remember, the visible destination IP address in the encrypted IP packet is 172.16.0.10 which is part of 172.16.0.0/24 with a next-hop attachment of FastConnect VC. That being said, the Loopback attachment route table will forward the encrypted packet over the FastConnect VC, good enough to call it TransportMode, right?

How the above routes arrives in the route table of the Loopback attachment? As we previously discussed, all properties of a Loopback attachment are read-only. The import route distributions as well as the route table will be read-only and system managed:

In the above picture we can see the route distribution-import associated with the Loopback attachment route table which automatically imports the FastConnect VCs received routes. There is no way for this one to be modified by an operator, it is system managed.

Finally, the traffic arrives at the 172.16.0.10 firewall where it is decrypted and sent to the on-premises host at 172.16.0.27.

Encrypted traffic from on-premises host at 172.16.0.27 to OCI VM at 192.168.1.3

This case analyze the on-premises response and well as new encrypted traffic originated from on-premises to OCI, the logic will is the same.

Once the 172.16.0.27 responds or initiate a new session to OCI at 192.168.1.3, the traffic will be encrypted by the on-premises firewall and send over the FastConnect VC. The traffic arrives at the DRG and it is using the FastConnect VC attachment route table. The FastConnect VC attachment route table contains the following entries:

It is enough to have only the routes for the OCI IPSec tunnels IP addresses? The answer is yes because the packet is encrypted and the visible IP addresses are only the IPSec head ends.

How the OCI IPSec tunnels IP addresses were added to the route table? We can find the answer by analyzing the import route-distribution associated with the route table:

The two entries were automatically added by the system for the DRG to be able to route the incoming encrypted traffic from on-premises to the IPSec service via the Loopback attachment, and second, for the DRG to announce the OCI IPSec tunnels IP addresses to on-premises via BGP.

The two above entries cannot be deleted and in TransportMode you will not be able to add any other distribution statements in the FastConnect VC route distribution-import. The TransportMode permits only IPSec traffic.

Added to the restriction for the FastConnect VC route distribution-import, if  the TransportMode is going to be configured, then OCI enforces some restrictions for the other attachments route distribution-import (except the associated Loopback attachment(s)) as follow: Match All or Attachment Type Virtual Circuit or Attachment Virtual Circuit where the TransportMode flag is set are not permitted to be configured. The same is true for static routes configured.

Note: If the TransportMode is configured and if any route distribution-import contains rules that violates the above restrictions, OCI will return an error and provides the information on how to remediate it.

Based on the route above, the encrypted traffic is sent to the Loopback attachment, then to the IPSec service for decryption and the decrypted traffic is looped back to the DRG on the IPSec attachment.

The IPSec attachment has the following route entries in its associated route table:

Now, the decrypted traffic is sent to the VCN based on the dynamic routes learned.

Let's prove the traffic is flowing encrypted performing a packet capture on the firewall at 172.16.0.10 once the traffic is generated from 192.168.1.3:

We can clearly see that the ESP traffic with the correct IP addresses is received on the firewall and the traffic flow works as expected.

2. All Traffic

What if the requirement is to send some traffic encrypted and some traffic non-encrypted over the FastConnect VC, is that possible? The answer is yes, the FastConnect VC can support the "All Traffic" configuration.

The "All Traffic" configuration will remove the restrictions defined above and will permit the configuration to distinguish between the encrypted vs non-encrypted traffic.

As in the previous configuration, we will analyze two use cases.

Non-encrypted traffic from OCI VM at 192.168.1.3 to on-premises host at 10.0.0.10

The networking diagram used for this use case is listed below:

The 10.0.0.10 should receive non-encrypted traffic from 192.168.1.3. Let's follow the path.

The first configuration we should perform is to configure the FastConnect VC for "All Traffic":

The traffic originated from 192.168.1.3 is reaching the DRG VCN attachment route table. The route table contains the following entries:

10.0.0.10/32 is directly reachable via the FastConnect VC attachment, it is not passing through the IPSec logic. As we can observe, 172.16.0.16/28 is still reachable via the IPSec.

We can clearly see that the DRG VCN attachment route table can import the FastConnect VC received routes (remember the restriction we had in the previous case).

Non-encrypted traffic from on-premises host at 10.0.0.10 to OCI VM at 192.168.1.3

The response is sent back from 10.0.0.10 and the traffic is traversing the FastConnect VC. The non-encrypted traffic arrives at the DRG and the DRG FastConnect VC attachment route table is used. Let's analyze the route table content:

The OCI VM is directly reachable via the VCN attachment. This means that we were able to dynamically import the route which actually means that the "All Traffic" FastConnect VC does not have the restriction to only import the Loopbacks.

Let's verify if indeed the traffic is sent in a non-encrypted form by configuring a packet capture on the on-premises firewall:

This proves the traffic for this network segment is not encrypted.

With this part, we finished our IPSec over FastConnect deep dive blog. I do have something special for you in my next blog, we will integrate the new feature with a Firewall within OCI to have all the traffic inspected, sort of Hub and Spoke type of approach with the IPSec over FastConnect in place.