Recently, I worked with a customer who wanted to create a global network in OCI based on their customer locations around the world. One major roadblock they encountered was route manipulation, specifically the challenge of controlling or influencing incoming and outgoing traffic to OCI. Additionally, managing traffic within the OCI network presented further complexities.

Before we dig deeper into the design portion there are few things that we should review about DRG and how it routes traffic.

Static routes always take precedence over dynamic routes. Conflicts between dynamic routes are resolved by first analyzing the route’s AS Path. If the conflict persists, the attachment type that imported the route is evaluated according to the following priority based on the attachment type:

1. VCN: the DRG makes an arbitrary but stable selection.
2. VIRTUAL_CIRCUIT: If ECMP is disabled, the DRG makes an arbitrary but stable selection. If ECMP is enabled, all routes will be added to the route table and the DRG makes routing choices using ECMP. The maximum supported ECMP width inside a DRG is 8.
3. IPSEC_TUNNEL: If ECMP is disabled, the DRG makes an arbitrary but stable selection.If ECMP is enabled, all routes will be added to the route table and the DRG makes routing choices using ECMP. The maximum supported ECMP width inside a DRG is 8.
4. REMOTE_PEERING_CONNECTION: The DRG will choose the route with the lowest network distance.

This last rule is crucial and will be the primary focus of this blog. It is significant for global design discussions because one of the fundamental building blocks in designing global connectivity is using RPC to connect multiple regions.

DRG treats routes advertised over RPC as if they were advertised over an iBGP session, meaning no BGP attributes are modified. Therefore, when designing a Full Mesh or Spine-Leaf topology, the routes received on all the DRGs will be handled consistently across all DRGs in the path.

Consider this example where on-Prem route 10.0.0.0/8 is being advertised to the DRG in Santiago region. Based on the topology the route will be advertised from both regions i.e. Vinhedo and Phoenix, since total latency/distance to reach the 10.0.0.0/8 will be less from Vinhedo so the route from Vinhedo will be accepted and installed in the Santiago DRG.

This route selection criteria by DRG make the Global connectivity design option logical since traffic will always take the lower latency path if other routes are same that that being evaluated by the route selection criteria.

To explore different design options and how OCI controls traffic through its backbone, let’s take an example of a customer X. Customer X wants to deploy OCI workload in multiple locations. They also want to connect multiple partners in different regions using IPSec and FastConnect. One limitation is that they don’t have the ability to influence traffic coming to them so no AS_PATH prepend support.

Lets consider these two simple architecture options

Full Mesh Topology

This topology will provide the best latency between the sites. One thing to note here when FastConnect routes are exchanged between the sites over PRC for example in this case between Santiago and Valparaiso DRG will not do any route manipulations. One should consider the route being advertised the over the RPC as routes being advertised over an IBGP peering (DRG will NOT do an AS_PATH prepend when the routes are exchanged between the region). For any region receiving the routes from remote region will select Santiago or Valparaiso based on the latency, whichever region will have better latency that route will be accepted. One of the downsides of having a full mesh topology is scalability because adding a new site will require r=n*(n-1)/2 (where r is no of RPC connection and n is the number of regions) RPC connection. If the number of regions grow the number of RPC needed will grow exponentially. This leads us to our next design option.

Spine Leaf Architecture

This option provides a balance between number of RPC connection needed and achieving optimal latency through the OCI backbone to reach remote destination over IPSEC and FastConnect location. The main idea here is to use the tried and tested Spine leaf architecture and have the DRG pick the best next hop based on latency in OCI network.

Conclusion

The goal of this discussion is to illustrate how customers can leverage OCI’s latency-based routing to achieve optimal latency within the OCI network. While I won’t delve into the specifics of configuring these architectures on OCI—saving that for a future blog—this overview will provide valuable insights into how cross-region routing can be effectively implemented using OCI networking constructs.