FastConnect Public Peering: Architectures and Use Cases

Introduction:

FastConnect public peering is a type of virtual circuit that allows customers to access public services in Oracle Cloud Infrastructure over your leased physical connection instead of using the internet.

By default, when you connect with FastConnect to Oracle Cloud Infrastructure (OCI) in a particular region, the routes advertised over the public virtual circuit include routes for other OCI regions in the same market, and for specific Oracle Cloud Infrastructure Classic regions.

Route filtering allows you to select what routes are included in BGP advertisements to your on-premises network. Route filtering options are:

• Regional - Advertises only available public routes for this virtual circuit's region to your on premises network.
• Market - Advertises available public routes for this virtual circuit's region and routes for other regions in the same part of the world to your on-premises network. This is the default setting.
• Global - Advertises available public routes for all regions in all markets of the Oracle cloud to your on-premises network.
• Oracle Services Network - Advertises only public routes used to access Oracle Services Network (OSN) resources to your on-premises network.

When customers have multiple FastConnect public peering connections, it becomes necessary to understand the routing behavior. In this blog, we will learn routing preferences when multiple public peering circuits are deployed on OCI.

We will explore 2 use cases which use multi region deployment of FastConnect public peering:

1.     Network Scenario 1: 2 Virtual Circuits with different route filtering

In this scenario, we will review the set up where 2 public peering virtual circuits with distinct route filtering are deployed in 2 different regions. We will then analyze it for the failover scenarios and come up with a design to ensure appropriate redundancy.

2.        Network Scenario 2: Multiple virtual circuits with different BGP parameters in different regions

In this scenario, we will make the design more complex where multiple public peering virtual circuits with different BGP attributes are deployed in different regions. We will analyze it for the failover scenario to fully understand OCI’s routing preference.

Network Scenario 1: 2 Virtual Circuits with different route filtering

In this scenario, we have 2 FastConnect public virtual circuits:

1. FastConnect 1 with Market route filtering terminating in Frankfurt region.
2. FastConnect 2 with Global route filtering terminating in Ashburn region.

London comes under market routes for Frankfurt and Singapore under global routes for Ashburn region. They are shown the diagram for a representation.

As a result, on premises location receives all the market level routes from Frankfurt region and global level routes from Ashburn region as shown in the diagram.

Both the virtual circuits terminate in the same on-premises location and on-premises advertise same public IP range on both the virtual circuits.

Let us understand how OCI prefers routing.

Because of the same route advertisements on both the virtual circuits, OCI receives the same routes from both the virtual circuits.

OCI commercial regions are interconnected with each other internally. In this scenario, OCI learns the same route in 2 different ways:

1. External: Public IP range is being learnt from on-premises on FastConnect 1 and OCI treats it as an external route.
2. Internal: Public IP range is being learnt by FastConnect 2 in Ashburn and then being internally learnt by FastConnect 2 in Frankfurt region. OCI treats it as an internal route.

By design, when choosing the return path, OCI always prefers external route over internal.

Hence, if the traffic is coming from Frankfurt virtual circuit, the return path will be same.

Thus, we can conclude that when multiple public peering virtual circuits with same BGP metrics are in place, OCI will always send the return traffic on the same virtual circuit, provided that the same route is being advertised from on-premises.

Now, let’s analyze this design for the failover scenario where one of the virtual circuits fails.

Failover Scenario 1: FastConnect 1 fails.

When FastConnect 1 fails, on premises will send traffic destined towards Frankfurt to the FastConnect 2 because of the global routes being received. Return path will be on the FastConnect 2 too as established previously in the blog.

Failover Scenario 2: FastConnect 2 fails.

In this scenario, for the traffic destined towards Ashburn and Singapore, FastConnect 1 cannot be used as a failover because of its Market level route filtering. This prevents on premises location from learning Ashburn and Singapore routes and traffic will not failover. Traffic towards Frankfurt and London will continue to be on FastConnect 1.

How do we make this design properly redundant?

Network Scenario 1: Solution to fix redundancy

The answer is to have both the virtual circuits set to global route filtering as follows:

In this design, we have two virtual circuits with global route filtering as shown. As all the parameters are identical (routes advertised/received, route filtering), OCI by default choses oldest established path as a tie breaker for the return path. It is always recommended to use BGP metrics to have better control over routing. Hence, in this case, let us suppose that FastConnect 1 is made primary with the help of AS path prepending. As a result, FastConnect 1 will be used to reach out all the regions from on premises and vice versa.

Failover scenario: FastConnect 1 fails.

In this scenario, both the virtual circuits can be used as redundant connections to each other as same routes are being advertised from OCI, as shown in the diagram. Hence generally speaking, we can conclude that to make the two public peering circuits redundant to each other:

1. Both the circuits need to be set to same route filtering.
2. Both the circuits should terminate in the region from the same market.

Thus, we have made the design fully redundant.

Now, let’s go further and see one more interesting scenario where multiple public peering circuits are used along with the BGP metrics configured.

Network Scenario 2: Multiple virtual circuits with different BGP parameters in different regions

In this scenario, in addition to the set up in Scenario 1, we have an additional virtual circuit FastConnect 3 with AS path prepending of x2, with the aim that the traffic will failover to the FastConnect 3 when FastConnect 1 fails as both the FastConnect circuits terminate in the same region and set to same route filtering. Routes sent over FastConnect 2 are not appended.

Now, as established previously, OCI will send the return traffic on the same virtual circuit it receives from.

Failover scenario: FastConnect 1 fails.

With the failure of FastConnect 1, traffic towards Frankfurt is being sent on FastConnect 3 now as per the requirement (this can be achieved with the help of BGP metrics like local pref). Now we expect that the return path would the same following the rules we set at the end of Network Scenario 1.

However, when FastConnect 1 fails, traffic still fails over to the FastConnect 2 by following the rule of shortest AS path. The reason being there are no prepends from on premises on FastConnect 2. Hence, OCI does not prefer routes its learning from FastConnect 3. This may result in asymmetric routing where, in the event of FastConnect 1 failure, customers may send the traffic on FastConnect 3 but return traffic will be sent on FastConnect 2.

We can conclude that shortest AS path always takes precedence over all the other parameters, including routes learnt externally. To summarize, we can say that in terms of OCI’s routing preference: shortest AS path > External routes > Internal routes.

Conclusion:

In this blog, we explored various network scenarios where multiple FastConnect public peering circuits are implemented along with different route filtering and BGP parameters. Customers must take all these scenarios into consideration before designing the network connectivity.