Introduction

 

This blog continues the discussion from a previous insightful post on multi-region FastConnect and VPN redundancy, which you can find here.

In this blog, we will explore advanced strategies for establishing redundant connections, featuring multiple FastConnect links within each region and using Remote Peering Connection (RPC) as a backup of last resort. This approach will provide deeper insights into traffic path selection when integrating these connectivity options.

For the scenario involving one FastConnect in each region, please refer to Part 1 of this blog, which can be found here.

A robust and reliable network infrastructure is crucial in today’s digital landscape. By employing multiple FastConnect and VPN connections across each region, we can significantly enhance the redundancy and resilience of our network architecture. This strategy mitigates the risk of potential failures and optimizes the performance and availability of mission-critical applications.

In this blog, I will outline best practices for implementing these redundant connections, highlight considerations for seamless integration, and share insights from real-world use cases.

First, let’s examine the DRG (Dynamic Routing Gateway) routing decision-making process outlined in the official Oracle documentation. When a DRG needs to determine the routing path for a subnet that is being dynamically accessed through various attachment types, the routing priority is as following:

1. VCN (Virtual Cloud Network): This attachment is prioritized first, as it offers a direct connection within Oracle’s cloud infrastructure, providing optimal performance and reliability.
2. VC (Virtual Circuit): This option is chosen next, leveraging Oracle FastConnect for a dedicated, high-bandwidth connection to your on-premises network or other cloud environments.
3. IPSec: This secure VPN tunnel is the third choice, offering encrypted connections over the public internet, making it suitable for secure data transmission when direct connections are unavailable.
4. RPC (Remote Peering Connection): This option is selected as a last resort. It provides peering connectivity between different regions or tenancies, adding an additional layer of redundancy.

Understanding this routing order is essential for designing a comprehensive network strategy that maximizes efficiency and ensures business continuity by effectively leveraging the strengths of each attachment type.

In addition to the DRG’s attachment order, it will also adhere to well-established routing rules within the routing table:

1. Static Route Precedence: Static route rules take precedence over dynamic route rules. This means that manually defined routes will always be selected over those learned dynamically, providing administrators with greater control over routing paths.
2. Longest Prefix Match: This principle states that the route in a routing table with the most specific subnet mask (the longest prefix) will be preferred among multiple potential routes. This ensures that traffic is directed along the most specific path available, enhancing routing accuracy and efficiency.

 By keeping these routing table rules in mind, you can design your network more effectively, ensuring optimal data flow and meeting your organization’s connectivity needs.

It’s important to understand that the prefixes advertised by OCI over BGP do not include any AS Path Length on any of the advertisement links. This is true even for FastConnect links that are not in the same region as the Virtual Cloud Network (VCN). Given this information, please ensure that the on-premises Customer Premises Equipment (CPE) is properly configured for BGP so that the traffic to OCI resources follows the desired path.

 

Prerequisites

 

Focus will be on achieving redundancy by utilizing one or multiple FastConnect connections per region, and implementing RPC as a last-resort backup. To effectively understand and apply these strategies, it is essential to familiarize yourself with key components of Dynamic Routing Gateway (DRG) Routing Management, as detailed in the official Oracle documentation.

This blog will not cover the creation of required components for end-to-end connectivity, such as VCNs, subnets, security lists, compute instances, DRGs, or other on-premises components.

We will not focus on configuring critical components that you already should know how to create, delete, modify, and utilize:

1. DRG Import Route Distribution: Understand how to manage the import of route distributions to ensure that your DRG dynamically learns and incorporates routes from various connections.
2. DRG Routing Tables: Learn how to configure and manipulate routing tables to define the rules governing how traffic is routed through your network. This setup includes establishing static routes and prioritizing them over dynamic ones.
3. DRG Attachments: Explore how to create and manage attachments to the DRG, including FastConnect connections, to ensure your network can effectively handle traffic from diverse sources.

By understanding these components and their functionalities, you will be better equipped to establish a resilient and strategically redundant network configuration that meets your organization’s needs.

As a prerequisite, it’s crucial to understand that a DRG routing table plays a vital role in managing incoming traffic for its respective attachments. For instance, when a DRG routing table is linked to a Virtual Cloud Network (VCN), it becomes instrumental in managing traffic leaving the VCN and reaching the DRG.

Here’s how this works:

1. Incoming Traffic Management: When traffic is directed from a Virtual Cloud Network (VCN), Virtual Cloud (VC), or Remote Peering Connection (RPC) to the Dynamic Routing Gateway (DRG), the DRG’s routing table determines how that traffic is managed and routed. It evaluates the defined routing rules to decide the next destination for the traffic, which could be on-premises networks (VC), other VCNs, or RPC.
2. Routing Logic: The routing table applies established rules, including the order of attachment types and the precedence of static versus dynamic routes. This ensures that the traffic follows the most suitable path based on your configurations.
3. Resulting Connectivity: By correctly configuring the DRG routing table for incoming traffic from various attachments, you can optimize connectivity and ensure that data flows efficiently to its intended destination.

 

Solution Description

 

Scenario 2 – Two FastConnect Links in Each Region: Phoenix and San Jose

If you prefer to view rather than read, check out this configuration video.

In this second scenario, as illustrated in the accompanying diagram, we have the following routing setup:

• On-premises network is advertising the IP range 10.0.0.0/8, which encompasses all addresses within that subnet. This range represents a significant portion of the IP space and serves as the primary entry point for traffic into the cloud infrastructure. In some real-world use cases I’ve encountered, customers typically choose to summarize their on-premises prefixes.
• Within the Oracle Cloud Infrastructure (OCI), the Phoenix region is configured with the subnet 10.1.0.0/16. This subnet is designated for resources hosted in that region, allowing for efficient traffic routing intended for these resources. Similarly, the San Jose region uses the subnet 10.2.0.0/16, which is designated for its own set of resources. This ensures a distinct and manageable addressing scheme for resources located in this regions.

In this scenario, with no changes made to the routing table from the previous setup, the Dynamic Routing Gateway (DRG) will arbitrarily select one Virtual Circuit (VC) as active while the other remains inactive. Here are the implications of this setup:

1. Since only one VC is selected as active, incoming and outgoing traffic may traverse different paths. This situation is known as asymmetric routing, where the route taken by the traffic to the destination differs from the route taken by the return traffic.
2. Asymmetric routing can lead to instability in network performance. If the Customer Premises Equipment (CPE) cannot effectively handle asymmetric traffic, it may result in packet loss, increased latency, and overall connectivity issues. Consequently, users may experience intermittent connectivity or degraded performance.

As an alternative to allowing asymmetric routing on the CPE—which is typically not permitted by security teams—you can activate Equal-Cost Multi-Path (ECMP) routing on the following routing tables: Customer Premises Equipment (CPE), Virtual Cloud Network (VCN), and Remote Peering Connection (RPC). Here’s how this approach can be beneficial:

1. ECMP Activation: By enabling ECMP on the CPE, VCN, and RPC routing tables, devices utilize a hashing mechanism per flow to determine the return traffic link. This ensures that outbound and inbound traffic consistently flows over the same connection.
2. Per-Flow Load Balancing: This configuration enables effective per-flow load balancing across both links. Traffic can be distributed evenly, optimizing resource usage and enhancing overall network performance. As a result, the combined bandwidth of both links can be utilized, providing greater throughput.
3. Advantages:
   • Increased Bandwidth: By leveraging both links, you maximize the available bandwidth, benefiting applications that require higher data rates.
   • Improved Redundancy: This configuration maintains redundancy; if one link fails, traffic can continue through the remaining link without major disruptions.
   • Consistent Routing Paths: ECMP helps prevent issues related to asymmetric routing by ensuring that traffic returns via the same path it took when initiating communication.
4. Disadvantages:
   • Configuration Complexity: Some CPE devices can be challenging to configure for ECMP. It may require advanced networking knowledge and careful attention to detail for proper implementation.
   • Potential Compatibility Issues: Not all devices support ECMP adequately, which might lead to inconsistent behavior in traffic handling.

To ensure that Equal-Cost Multi-Path (ECMP) routing functions correctly, the BGP AS Path Length must be consistent across all FastConnect links. This uniformity allows for balanced traffic distribution across the available links. If the AS Path Lengths are not the same, ECMP will not effectively utilize all paths, which could result in imbalanced traffic flow.

Once ECMP is enabled on Oracle Cloud Infrastructure (OCI), the links and routing tables will have the following configuration:

The VCN and RPC routing tables will show both VC links as active in this configuration.

 

 

 

Here are the key implications and advantages of this setup:

1. Active Dual Links: With both VC links marked as active in the routing tables, the network can effectively use both connections simultaneously, enhancing overall network resilience.
2. Load Balancing via ECMP: Enabling ECMP allows for even distribution of traffic across both active links, maximizing bandwidth usage and preventing any single link from becoming a bottleneck, thus improving performance.
3. Improved Network Reliability: If one of the links encounters issues or goes down, the other link can seamlessly handle the traffic flow without losing connectivity, significantly reducing the risk of downtime.
4. Consistent Traffic Management: With both links actively participating in traffic management, the routing tables can better manage routing decisions based on real-time conditions, ensuring optimal path selection.

If one of the FastConnect links goes down, its route will be removed from the VCN and RPC routing tables, and only the remaining active FastConnect link will be utilized, as illustrated in the following diagram:

 

Scenario 3 – Configuring Two FastConnect Links in Each Region with BGP AS Prepend for Primary and Secondary Virtual Circuits

If you prefer to view rather than read, check out this configuration video.

In this third scenario, as illustrated in the accompanying diagram, we have the following routing setup:

• On-premises network is advertising the IP range 10.0.0.0/8, which encompasses all addresses within that subnet. This range represents a significant portion of the IP space and serves as the primary entry point for traffic into the cloud infrastructure. In some real-world use cases I’ve encountered, customers typically choose to summarize their on-premises prefixes.
• Within the Oracle Cloud Infrastructure (OCI), the Phoenix region is configured with the subnet 10.1.0.0/16. This subnet is designated for resources hosted in that region, allowing for efficient traffic routing intended for these resources. Similarly, the San Jose region uses the subnet 10.2.0.0/16, which is designated for its own set of resources. This ensures a distinct and manageable addressing scheme for resources located in this regions.

To implement the scenario effectively, you need to make the following modifications:

1. Change the CPE Advertised prefix

• Action: Modify the prefix advertised by the CPE from a single /8 (10.0.0.0/8) to two /9 prefixes (10.0.0.0/9 and 10.128.0.0/9).
• Purpose: This change segments the network, allowing the Virtual Circuits to be preferred over a static summarized prefix of 10.0.0.0/8.

2. Add BGP AS Prepend on the Secondary FastConnect Link  

• Action: On the CPE, apply an AS Prepend of 1x on the secondary FastConnect link’s prefix advertisement to OCI.
• Purpose: This configuration enables OCI routing tables to recognize the primary and secondary setup of the FastConnect connections.

3. Modify VCN Import Route Distribution

• Action: Change the VCN routing table’s import distribution from a “Match All” rule to import only those attachment types relevant in this scenario:
  • Virtual Cloud Network
  • Virtual Circuit
• Purpose: This modification ensures that the routing table only considers routes relevant to the VCN and VC attachment types, preventing automatic import of routes from the Remote Peering Connection. 

4. Add Static Routes Over RPC

• Action: Create static routes in the RPC routing table for the following subnets:
  • On-Premises Subnet: 10.0.0.0/8
  • San Jose Region Subnet: 10.2.0.0/16 in Phoenix
  • Phoenix Region Subnet: 10.1.0.0/16 in San Jose
  • Destination via: Remote Peering Connection
• Purpose: Adding these static routes ensures that traffic intended for regional subnets is accurately routed through the RPC. Additionally, traffic destined for on-premises (10.0.0.0/8) will be routed via the RPC only if both regional Virtual Circuits go down.  

Once these modifications are implemented, the routing tables should reflect the changes, showcasing the following configurations in your diagram:

1. VCN Import Route Distribution and Routing Table:

 

2. RPC Import Route Distribution and Routing Table:

 

3. Virtual Circuit Import Route Distribution and Routing Table:

 

In the resulting routing tables, VC-1 will be installed, while VC-2 will be uninstalled and marked as “Conflict,” since VC-1 is designated as the primary link and VC-2 is the secondary link.

In this scenario, the network architecture adjusts when the primary VC fails, in this case, in the Phoenix region. The subsequent changes will be reflected in the VCN and RPC routing tables:

In this updated scenario, after the failure of the VC-1 link in the Phoenix region, you will observe the following modifications in the VCN and RPC routing tables:

1. VCN Routing Table:The VCN routing table will now only display the availability of VC-2. The entry for VC-1 will be removed, ensuring that only the operational link is used for routing traffic. This adjustment prevents any attempts to route traffic through the failed VC-1 link, thereby maintaining efficient traffic management.

2. RPC Routing Table: Similarly, the RPC routing table will reflect that only VC-2 is available for routing. This ensures that routes dependent on the RPC connection effectively utilize the available links. It may also show active static routes directing traffic to the appropriate destinations in both on-premises and regional subnets.

In conclusion, the DRG is a powerful service that enhances the security and resilience of private connectivity to OCI. This exploration of various scenarios demonstrates how leveraging multiple FastConnect connections and implementing routing strategies like ECMP and BGP AS Prepend can optimize network performance and reliability.

By understanding the capabilities and configurations of the DRG, organizations can design robust networks that adapt to changing conditions, ensuring continuous access to vital resources without compromising security.

Thank you for reading, and happy networking!