Introduction

Caveats and Limitations

Prerequisites

Step 1. Creating RPC link for Customer 1 in AMS

Step 2. Creating an LPG and IAM Policy for Customer 2 in ZHR

Step 3. Creating an RPC link for Customer 1 ZHR

Step 4. Creating an IAM Policy and LPG link for Customer 1 ZHR

Connectivity Validation

Introduction

Oracle cloud offers multiple network technologies that allow you to connect different networks together. Usually we see these two use cases:

1. The networks are in the same tenancy but in different OCI regions,
2. or they are in the same OCI region but in two different tenancies.

From time to time we see a third (3rd) use case appear which is:

3. Two different tenancies that need connectivity across two different regions. Such as Company 1 in Amsterdam needs access to Company 2 in Frankfurt

Since I get many questions regarding the possibility of interconnecting two tenancies in separate regions, I have decided to make this document with the purpose to describe how to achieve this in OCI by leveraging Remote Peering Connection (RPC) connectivity and Local Peering Gateway (LPG) connectivity with transit routing enabled.

In this blog we will also add some Identity and Access Management (IAM) policies, that is needed for creating intra-tenancy LPG’s (Customer 1 ZHR VCN and Customer 2 ZHR VCN). IAM Policies lets you control who has access to your cloud resources and in this case will let Customer 1 to be the requestor of the LPG peering and Customer 2 the Acceptor of the LPG Peering.

For more information regarding:

• RPC please see Oracle documentation at https://docs.cloud.oracle.com/en-us/iaas/Content/Network/Tasks/remoteVCNpeering.htm
• LPG please see Oracle documentation at https://docs.cloud.oracle.com/en-us/iaas/Content/Network/Tasks/localVCNpeering.htm
• IAM Please see Oracle documentation at https://docs.cloud.oracle.com/en-us/iaas/Content/Identity/Concepts/overview.htm

Caveats and Limitations

1. RPC connectivity is not possible between 2 different tenancies, since RPC is used to create a connectivity link between 2 regions VCN’s using DRG. This link is not designed to be used for On-Prem connectivity to reach another region VCN, and private IP traffic from On-Prem -> AMS VCN ->RPC-> ZHR VCN will not work and a direct link to region is needed.
2. Connectivity will be done over Oracle backbone,  backbone that is monitored for capacity and latency by OCI, there will be no bandwidth or latency SLA on this link, and because of that any business-critical traffic should not use this path, but instead a dedicated Fastconnect link via a provider network that can provide proper SLA’s.
3. For an RPC, it is required to have at least a single active tenancy in multiple regions such as Frankfurt and Amsterdam. RCP can be created between any regions and is not limited by a geographic region like NA or EMEA

4. In addition, for the LPGs, it is required to have two active VCNs in the same region where the first tenancy will act as a HUB tenancy. An LPG can be created between 2 VCN’s in same tenancy or between 2 VCN’s in different tenancies with the condition that VCN’s are in same region

Prerequisites:

1. In this document I will not concentrate on building the infrastructure elements like Virtual Cloud Network (VCN), Dynamic Routing Gateway (DRG), Routing Table, Security List (SL), Network Security Group (NSG) and Virtual Machine (VM), we will assume that all these elements are already in place.
2. Identity and Access Management (IAM) policies for allowing work in same tenancy is considered as already existing, for example to add/update Routing Tables, SL/NSG, RPC, IAM policy etc.
3. Validating VCN subnets are not overlapping on all VCN’s involved, in this example I have used following subnets:

1. Customer 1 VCN in AMS - 192.168.100.0/24

     Customer 1 VCN in ZHR – 192.168.1.0/24

2. Customer 2 VCN in ZHR – 172.16.0.0/24

Solution description.

In order to overcome the limitation that RPC is available only on the same tenancy I have created following path as an example

Customer 1 AMS--> RPC to Customer 1 ZHR --> LPG to Customer 2 ZHR as in the following diagram

For this solution to work end to end we still need to have some SL updated as we can see in next diagram

Step 1. Creating RPC link for Customer 1 in AMS

1. Update the Routing Table of Customer 1 AMS VCN of the subnet that is part of the source or destination (see below diagram), with Customer 2 ZHR VCN subnet 172.16.0.0/24 for required connectivity with a route using the DRG as net hop

After I have done this, I have in my Customer 1 AMS following route rules, that include 172.16.0.0/24 via attached DRG

2.  Update SL/NSG with the needed rules to allow required traffic, in this case I have updated SL from the following diagram

with following rules

Ingress

Egress

3. Creating RPC link:

1. Go to Customer 1 DRG in AMS that is attached to the VCN that has the VM needing to communicate with Customer 2 VCN - > Remote Peering Connections -> Create Remote Peering Connection
2. Fill the name field and select wanted compartment and hit Create Remote Peering Connection

3. After doing that a new field will appear with this new peer created

4. Select 3 dots menu in the right and Copy Remote Peering Connection OCID of the created peering

With this step the work is done for Customer 1 AMS region.

Step 2. Creating an LPG and IAM Policy for Customer 2 in ZHR

1. Create IAM policy to allow customer 1 to establish a LPG by adding a policy under Identity -> Policies

Create Policy in root compartment with following statement for Acceptor tenancy, in this case Customer 2

Define tenancy [Requestor] as „ocid1.tenancy.oc1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”

Define group [Requestor Group] as „ocid1.group.oc1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”

Admit group [Requestor Group] of tenancy [Requestor] to manage local-peering-to in compartment maradule

Admit group [Requestor Group] of tenancy [Requestor] to associate local-peering-gateways in tenancy [Requestor] with local-peering-gateways in compartment [name of the compartment]

2. Create LPG

1. Go to VCN -> Local Peering Gateways -> Create Local Peering Gateway, fill the name of the Link and select the compartment and hit Create Local Peering Gateway

2. After doing this a new connection will appear the new LPG

3. Go to 3-point menu in the right and select Copy OCID

3. Update Routing Table of the subnet that is part of the source or destination for required connectivity with a route using the new created LPG as net hop

4. Update SL/NSG with the needed rules to allow required traffic. In my example I have update with following rules

Ingress

Egress

Step 3. Creating an RPC link for Customer 1 ZHR

1. Creating and activating RPC link

1. Create RPC following same steps as we did above on Step 1 Point C up to Section III
2. Activate RPC by selecting Establish Connection from 3-point menu in the right
3. Select correct region where we want to connect, in this example will be eu-amsterdam-1, and fill in the OCID of the RPC copied on Step 1 section IV and hit Establish Connection button

After this is done connection should look like

Things to take into consideration about RPC:

• RPC are attached to a DRG that is attached to VCN
• RPC will take some time before it's fully established
• RPC is peered when Peering status is “Peered”
• A traceroute over an RPC will show only DRG IP’s not anything else since traffic is encapsulated in MPLS labels, but it can be determined since from one DRG to another DRG we will see a delay increase depending on the distance (check traceroute in “Connectivity Validation” section).

Step 4. Creating an IAM Policy and LPG link for Customer 1 ZHR

1. Create IAM policy to allow Customer 1 to establish an LPG by adding a policy under Identity -> Policies (see Step 2 Point A)

Create Policy in root compartment with following statement for Requestor tenancy, in this case Customer 1

Define tenancy [Acceptor] as „ocid1.tenancy.oc1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”

Allow group [Requestor Group] to manage local-peering-from in compartment [name of the compartment]

Endorse group [Requestor Group] to manage local-peering-to in tenancy [Acceptor]

Endorse group [Requestor Group] to associate local-peering-gateways in compartment [name of the compartment] with local-peering-gateways in tenancy [Acceptor]

2. Create LPG following same steps as we did above on the Step 2 Point B, up to Section II, and after establishing the LPG by going to 3-point menu in the right and select Establish Peering Connection

Select ENTER LOCAL PEERING GATEWAY OCID and paste the OCID of the peering copied at Step 2 Point B Section III and push Establish Peering Connection

3. Create/update DRG routing table by adding the Customer 2 VCN subnet (in this case 172.16.0.0/24) via LPG created at Step 4 Point B

1. Attach created routing table to DRG by going to VCN -> Dynamic Routing Gateway and here from 3-point menu from the right and select Associate Route Table (optional only if Route Table is not already associated, in this case update that route table). After this is done DRG should look like this example:

4. Create/update LPG routing table by adding Customer 2 AMS VCN Subnet (in this case 192.168.100.0/24) via DRG

1. Attach created routing table to LPG by going to VCN -> Local Peering Gateways and here from 3-point menu from the right and select Associate Route Table. After this is done LPG should look like this example:

And with this the configuration is done.

Connectivity Validation

For verifying connectivity, I have the following diagram:

I have created a VM on Customer 1 AMS with following interface configuration

I have created a VM on Customer 2 ZHR with following interface configuration

I have done a ping from Customer 1 AMS VM to Customer 2 ZHR VM

I have done a TCPDUMP on Customer 2 ZHR VM during ping

In addition, I have opened Ingress UDP ports in Customer 2 ZHR SL (same as Step 2 Point D)

so, now I can do a traceroute and a trace path from Customer 1 AMS VM to Customer 2 ZHR VM

As we can see from Customer 1 AMS VM (192.168.100.2) a traceroute to Customer 2 ZHR VM (171.16.0.2) we can see following data:

• AMS DRG (140.92.222.63 and 140.91.222.33)
• ZHR DRG (140.91.210.9)
• Customer 2 ZHR VM (172.16.0.2)

We can see OCI internal DRG IP’s as this is the exiting interface IP and it is expected to be seen since those devices are part of the internal backbone connectivity, and it is an expected behavior. 

Also, between hop 1 and 2 we can see there is a latency increase of about 23 ms, which is the latency between AMS region and ZHR region, expected round trip delay for geographical distance which is about 850 Km. For example a latency between AMS region and IAD (Ashburn) region (about 6000Km), a typical latency is about 110 ms. This must be taken in to consideration since latency is one of the values that is determining maximum throughput of a link.

Thank you.