Best Practices from Oracle Development's A‑Team

ISV Architecture Validation - Implementation Details - Part 3B - OCI vRouter Instances

Tal Altman
Sr. Manager

This is the 3rd blog, (Part 3B), in a series of blogs regarding the ISV Validated Design. 

This blog series will contain the following topics

  1. ISV Home Page
  2. ISV Architecture Validated Design 
    • Requirements, Design, Solution
    • Life of a packet
    • High Availability (HA) Concepts
  3. Core Implementation
  4. Failover Implementation – You can choose from the two options below for implementation
  5. Operations
    • How to add a customer to an existing POD
    • How to create a new POD
    • References, key files and commands




In this blog we will focus on the OCI Instances and how to configure them to support the large scale design for ISV's. In the last blog entry we focused on the networks, route tables, and security lists needed. Now we will focus on how to setup the virtual router instances.

Virtual Routers and network topology


Design guidance on the Instance shape

  • Think about the quantity of PODs, and the total bandwidth you need between your Management networks and your End Customer instances. Choose an Instance shape that is appropriate for the bandwidth you'll need, the quantity of VNICs, etc.

  • PODs. 

    • By default your POD network can peer with up to 10 Customers networks. With a service limit request, you can increase that limit from 10 to 20.

  • Bandwidth

    • Each customer should have their own Dynamic routing gateway or Internet Gateway, so in theory most of the customer traffic should go out the DRG, or Internet Gateway.

    • Most traffic should be "control plane" traffic (from Customer instances to Management Instances), versus "data plane" traffic (i.e. Customer 1 accessing their application running in the Customer 1 VCN over the Internet/VPN, etc).

  • You can change the Instance shape if needed, but it might require stopping the instance.



Virtual Router Instance Setup in OCI Console

Virtual Router Instance details

  • To Launch an instance you have to make some design choices regarding the OS, Hardware Resources, AD or Fault Domain placement, etc. Here are some screen shots from the OCI Console.
  • I recommend using Oracle Linux 7.x or using Oracle Autonomous Linux.
  • Make sure your routers are in different fault domains or Availability domains for Disaster recovery purposes....
  • Subnets should be regional so your network can span multiple ADs/Fault Domains.

Screenshot - Creating a new compute instance. Make sure to give your router a name!


Screenshot - Name your instance


Screenshot - If you click on "Change Image Source, and then click on the "Oracle Images" section you can deploy the Oracle Autonomous Linux instance.

Screenshot - Browse All Images

Screenshot - Pick an Availability Domain, Instance Type and Shape


Screenshot - Select an AD, Instance type and shape


Screenshot - Choose a shape. Review the Network Bandwidth (aggregate across all interfaces) and the Total VNICs (or PODs you can attach to). For example if you select the 2.16 shape, you'll have 1 primary VNIC, and you can attach up to 15 PODs.


Screenshot - pick a shape


Screenshot - Attach the primary VNIC to the vrouters subnet

Screenshot - Virtual networking


Screenshot - Choose a Fault Domain. Make sure to place subsequent routers in different Fault Domains.

Screenshot - pick a Fault Domain




Virtual Router Instance - Network details


1 - Virtual Router Instance - Enable "Skip Source/Destination Check"

  • Each virtual router should have at least two VNICs
  • Each virtual NIC should have skip-source/destination check enabled
    • Go to Instance → VNIC → Edit VNIC → add the check mark next to "Skip Source/Destination Check."

VNIC details



2 - Virtual Router Instance - Create a second VNIC and attach it to the 1st POD network 

  • Create additional VNICs to attach to each POD. The second vNIC in our example will connect to POD1.
    • Go to Instance → Click on "Attached VNICs" and then "Create VNIC"
      • Give the VNIC a name, such as "vRouter#-pod#"
      • Choose a VCN, and subnet to attach to. Such as POD1 VCN, and vnic-subnet (regional).
      • Enable Skip Source/Destination Check also
      • Statically assign the IP Address of this VNIC.

Second VNIC on the 1st POD network


3 - Virtual Router - Secondary or "Floating" IP Addresses

  • For High Availability we need to have a Floating IP address that can bounce around from one vRouter to another depending on the availability of the nodes. In physical networking this would be accomplished with HSRP, or VVRP. In Cloud networks, we utilize a secondary IP address for this concept. Lets assume vRouter1 and vRouter2 are on the same network and share a common Floating IP address of If vRouter1 fails, then a watchdog service on vRouter #2 could tell the OCI Console to un-assign the floating IP address from vRouter1 and assign it to vRouter2. OCI's console has the ability to un-assign an IP address if its already assigned to another vNIC. Now we don't want to affect the primary IP address of the host. We want to adjust the second IP address so it doesn't affect the availability and reachability of vrouter1 or vrouter2. 
  • We utilize a technique of adding a second IP address to an existing VNIC to accomplish this task.

Assign Private IP


  • To create a Floating IP Address, we are going to assign a second IP address in the OCI Console, to the Primary VNIC, and the POD VNIC on vRouter1
  • In the HA Configuration section of this document we will go into more detail about how to move the Floating IP Addresses from one host to another.

Note: The Secondary IP address should only be defined once in the OCI Console per network. You'll need to configure it on each and every linux instance, but from an OCI Console point of view only one instance can have the Secondary IP address.


  • To configure...Go to the Instance Details of vRouter1 → click on Attached VNICs → Click on the Primary VNIC → Click on IP Addresses. 
  • In this location you'll be able to see the current IP assignments for this VNIC. There is a button that you can click on called "Assign Private IP Address", go ahead and click on that.
  • Breadcrumbs: Compute >> Instances >> Instance Details >> Attached VNICs >> VNIC Details >> IP Addresses

VNIC Details with multiple IPs




Linux configuration


This section was validated and tested on Oracle Linux 7.x, and should work on most RedHat/CentOS based distributions.


Note: The commands or files utilized might vary from one distribution to the next. All commands below assume you have performed a "sudo sh", or "sudo bash" so you have root privileges to run these commands.


To setup your Linux instance as a dual homed router, I would recommend taking the following steps:

1) Log into the host via SSH

2) Verify the network interface label for the new VNIC

When you add a second VNIC interface (or a 3rd, 4th, etc), LInux will give it a name such as "ens5". To verify what the new label name is, download and run a script from Oracle.com. In the example below you can see the new "IFACE" is ens5.

Example commands

wget https://docs.cloud.oracle.com/iaas/Content/Resources/Assets/secondary_vnic_all_configure.sh
chmod a+x secondary_vnic_all_configure.sh


Example of the script executing:


3) Assign the IP addresses to the network cards and update the MTU to 9000 (for Jumbo frames)


ip addr add dev ens3 label ens3:0
ip link set ens5 mtu 9000
ip addr add dev ens5 label ens5
ip addr add dev ens5 label ens5:0


4) Create an Interface Configuration startup script for each VNIC and its floating IP addresses.

Your virtual router will have at least 4 interface startup scripts.

/etc/sysconfig/network-scripts/ifcfg-ens3 (primary-vnic)
/etc/sysconfig/network-scripts/ifcfg-ens3:0 (primary-vnic secondary IP)
/etc/sysconfig/network-scripts/ifcfg-ens5 (pod1-vnic)
/etc/sysconfig/network-scripts/ifcfg-ens5:0 (pod1-secondary IP)


The sample configuration for the startup script can be seen below. This will help ensure that the configuration is persistent upon reboots.

vi /etc/sysconfig/network-scripts/ifcfg-ens5


5) Add static routes so that the Linux host knows how to reach your customer networks.

Ensure that VNIC routing entries are persistent upon reboots. To make the static routes persistent you can create a route- startup script. See the example below.

vi /etc/sysconfig/network-scripts/route-ens5 via dev ens5 via dev ens5

6) Enable IP Forwarding so your Linux host will start forwarding traffic between interfaces.

vi /etc/sysctl.d/98-ip-forward.conf


7) We need to enable "loose" reverse path forwarding to enable use cases where asymmetric routing could occur. For example, traffic comes in on vrouter1, but comes back on vrouter2, etc.

vi /etc/sysctl.d/97-reverse-path-forwarding.conf

8) Restart the network:

systemctl restart network


Note: The command may fail, or you may see errors that a duplicate IP address exists. That is OK and can be safely ignored.



If you need to add static routes for testing purposes, here are a few example commands.

ip route add via dev ens5
ip route add via dev ens5
ip route add via dev ens3

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha