In this new blog post, we will discuss one of the very hot topics related to OCI DRGv2, the route conflict. The route conflict simply means the route or routes that are used by a particular DRG route table to forward the traffic when the same route is learned from multiple sources. We will not discuss ECMP since this subject has been fully covered in this blog post: https://www.ateam-oracle.com/equal-cost-multi-path-ecmp-routing-on-oci-drgv2.
We will analyze different use cases based on the route preference implemented, together with clear examples. The route conflict can be consulted in detail at this link: https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/managingDRGs.htm section "Route Conflicts".
As we know, we can have two types of routes in a DRGv2 route table, static and dynamic routes. In a nutshell, the Route Conflict is resolved using the following algorithm:
1. Static routes always have a higher preference than dynamic routes, the static routes will always have an empty BGP AS_PATH;
2. Conflicts between dynamic routes are resolved by first analyzing the route's BGP AS_PATH, the shortest AS_PATH always wins:
2.1 Routes with a route source of VCN always have an empty BGP AS_PATH;
2.2 Routes with a route source of IPSEC_TUNNEL or VIRTUAL_CIRCUIT will have BGP AS_PATH information;
3. Otherwise, the attachment type that imported the route is evaluated according to the following priority based on the attachment type: VCN > VC > IPSec > RPC;
4. If conflicting routes are imported from attachments of the same type, the conflict is resolved differently depending on the attachment type:
4.1 VCN Attachments: If identical CIDRs are imported from two VCN attachments, only one is selected;
4.2 VIRTUAL_CIRCUIT Attachments:
- If multiple routes with the same CIDR and different BGP AS_PATH lengths are imported into a DRG route table, the route with the lowest AS_PATH length is selected;
- If the BGP AS_PATH is the same and ECMP is enabled, all routes will be added to the route table for a maximum supported ECMP route of eight;
- If the BGP AS_PATH is the same and ECMP is disabled, one route is chosen;
4.3 IPSEC_TUNNEL Attachments: similar to VIRTUAL_CIRCUIT Attachments;
4.4 RPC attachments: If identical CIDRs are imported from two RPC attachments, one route is chosen;
Since some of the above points are quite easy to grasp, we will focus on some use cases to prove what is listed above is right.
We will focus on the 192.168.0.0/24 subnet received from multiple sources. We will analyze how the algorithm listed above picks the 192.168.0.0/24 and use it for traffic forwarding when it is received from multiple sources and inserted in a DRG route table. We will use the DRG route table attached to the VCN 1 from the above networking topology.
The DRG route table attached to VCN 1 uses a Route Distribution with the following Statements:
The DRG route table for VCN 1 imports 192.168.0.0/24 from BGP over IPSec Tunnel 1 attachment, from RPC attachment, and from VCN 2 attachment.
On another hand, the route table for VCN 1 attachment includes a static route to 192.168.0.0/24 with the next-hop attachment name being the VCN 3:
Let's have a look inside the DRG route table for VCN 1 attachment after all the above are configured:
As we expect, we have four entries for 192.168.0.0/24. The static route is marked as Active, the rest of the three dynamic routes received on different attachments are marked as Conflict routes. This confirms that the static route has a higher preference over dynamic routes.
Let's remove the static route from the DRG route table and check if the route using the VCN as a next-hop attachment becomes the Active one:
The static route disappeared from the route table and according to the algorithm listed above, the VCN route becomes the Active one.
To prove this use case we will use the same BGP AS_PATH for 192.168.0.0/24 announced to OCI from CPE 1 (Ashburn) and CPE 2 (Phoenix). That being said, the DRG from the Ashburn region will receive the 192.168.0.0/24 subnet with only one BGP AS in the AS_PATH from the customer side (as per the networking topology listed above). Remember that using BGP over IPSec at the receiving routes from the CPE side, OCI is prepending one more AS to the AS_PATH received from the CPE for a total of 2 AS in the BGP AS_PATH in our example. This action is documented at this link: https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/managingDRGs.htm.
The DRG from the Ashburn region will receive the same 192.168.0.0/24 over the RPC. In the Phoenix region, we are receiving the 192.168.0.0/24 subnet with the same BGP AS_PATH. Over RPC no AS prepending happens, so the DRGv2 from Ashburn will receive the 192.168.0.0/24 subnet with the same BGP AS_PATH.
Let's check the route table for VCN 1 attachment after removing the VCN route:
We can clearly see that the IPSec Tunnel next-hop attachment route is marked as Active due to equal BGP AS_PATH length and IPSec route being more prefered than the RPC route.
This use case sounds strange, right? In use case 3 we just proved that BGP over IPSec is more prefered than the RPC. Is this possible at all? Yes, it is possible. Remember, the first deciding factor for the dynamic routes is the BGP AS_PATH.
Let's prepend one time the customer BGP ASN on CPE 1 for the prefix received over the BGP on IPSec Tunnel 1 in Ashburn, for a final BGP AS_PATH of three (two from the customer side including the prepended one and one automatically added by OCI):
Now, let's verify the route table for VCN 1 attachment:
With a very simple configuration on CPE manipulating the BGP AS_PATH, OCI is sending the traffic to On-Premise for 192.168.0.0/24 over the RPC and not directly via the associated IPSec Tunnel in Ashburn.