This is the 3rd blog, (Part 3B), in a series of blogs regarding the ISV Validated Design. 

This blog series will contain the following topics

1. ISV Home Page
2. ISV Architecture Validated Design 
   • Requirements, Design, Solution
   • Life of a packet
   • High Availability (HA) Concepts
3. Core Implementation
   • Part 3A - OCI - Routing and Security Lists
   • Part 3B - OCI vRouter - OCI Shape, OS choices, VNIC and Secondary IP address creation
4. Failover Implementation – You can choose from the two options below for implementation
   • Part 4A - Linux Clustering with Pacemaker and Corosync
   • Part 4B - Linux Clustering with Keepalived (VRRP)
5. Operations
   • How to add a customer to an existing POD
   • How to create a new POD
   • References, key files and commands

Introduction

In this blog we will focus on the OCI Instances and how to configure them to support the large scale design for ISV's. In the last blog entry we focused on the networks, route tables, and security lists needed. Now we will focus on how to setup the virtual router instances.

Design guidance on the Instance shape

• Think about the quantity of PODs, and the total bandwidth you need between your Management networks and your End Customer instances. Choose an Instance shape that is appropriate for the bandwidth you'll need, the quantity of VNICs, etc.

• PODs. 

  • By default your POD network can peer with up to 10 Customers networks. With a service limit request, you can increase that limit from 10 to 20.

• Bandwidth

  • Each customer should have their own Dynamic routing gateway or Internet Gateway, so in theory most of the customer traffic should go out the DRG, or Internet Gateway.

  • Most traffic should be "control plane" traffic (from Customer instances to Management Instances), versus "data plane" traffic (i.e. Customer 1 accessing their application running in the Customer 1 VCN over the Internet/VPN, etc).

• You can change the Instance shape if needed, but it might require stopping the instance.

Virtual Router Instance Setup in OCI Console

Virtual Router Instance details

• To Launch an instance you have to make some design choices regarding the OS, Hardware Resources, AD or Fault Domain placement, etc. Here are some screen shots from the OCI Console.
• I recommend using Oracle Linux 7.x or using Oracle Autonomous Linux.
• Make sure your routers are in different fault domains or Availability domains for Disaster recovery purposes....
• Subnets should be regional so your network can span multiple ADs/Fault Domains.

Screenshot - Creating a new compute instance. Make sure to give your router a name!

Screenshot - If you click on "Change Image Source, and then click on the "Oracle Images" section you can deploy the Oracle Autonomous Linux instance.

Screenshot - Pick an Availability Domain, Instance Type and Shape

Screenshot - Choose a shape. Review the Network Bandwidth (aggregate across all interfaces) and the Total VNICs (or PODs you can attach to). For example if you select the 2.16 shape, you'll have 1 primary VNIC, and you can attach up to 15 PODs.

Screenshot - Attach the primary VNIC to the vrouters subnet

Screenshot - Choose a Fault Domain. Make sure to place subsequent routers in different Fault Domains.

Virtual Router Instance - Network details

1 - Virtual Router Instance - Enable "Skip Source/Destination Check"

• Each virtual router should have at least two VNICs
• Each virtual NIC should have skip-source/destination check enabled
  • Go to Instance → VNIC → Edit VNIC → add the check mark next to "Skip Source/Destination Check."

2 - Virtual Router Instance - Create a second VNIC and attach it to the 1st POD network 

• Create additional VNICs to attach to each POD. The second vNIC in our example will connect to POD1.
  • Go to Instance → Click on "Attached VNICs" and then "Create VNIC"
    • Give the VNIC a name, such as "vRouter#-pod#"
    • Choose a VCN, and subnet to attach to. Such as POD1 VCN, and vnic-subnet (regional).
    • Enable Skip Source/Destination Check also
    • Statically assign the IP Address of this VNIC.

3 - Virtual Router - Secondary or "Floating" IP Addresses

• For High Availability we need to have a Floating IP address that can bounce around from one vRouter to another depending on the availability of the nodes. In physical networking this would be accomplished with HSRP, or VVRP. In Cloud networks, we utilize a secondary IP address for this concept. Lets assume vRouter1 and vRouter2 are on the same network and share a common Floating IP address of 172.20.136.140. If vRouter1 fails, then a watchdog service on vRouter #2 could tell the OCI Console to un-assign the floating IP address from vRouter1 and assign it to vRouter2. OCI's console has the ability to un-assign an IP address if its already assigned to another vNIC. Now we don't want to affect the primary IP address of the host. We want to adjust the second IP address so it doesn't affect the availability and reachability of vrouter1 or vrouter2. 
• We utilize a technique of adding a second IP address to an existing VNIC to accomplish this task.

• To create a Floating IP Address, we are going to assign a second IP address in the OCI Console, to the Primary VNIC, and the POD VNIC on vRouter1

• In the HA Configuration section of this document we will go into more detail about how to move the Floating IP Addresses from one host to another.

• To configure...Go to the Instance Details of vRouter1 → click on Attached VNICs → Click on the Primary VNIC → Click on IP Addresses. 
• In this location you'll be able to see the current IP assignments for this VNIC. There is a button that you can click on called "Assign Private IP Address", go ahead and click on that.

• Breadcrumbs: Compute >> Instances >> Instance Details >> Attached VNICs >> VNIC Details >> IP Addresses

Linux configuration

This section was validated and tested on Oracle Linux 7.x, and should work on most RedHat/CentOS based distributions.

To setup your Linux instance as a dual homed router, I would recommend taking the following steps:

1) Log into the host via SSH. 

2) Verify the network interface label for the new VNIC

When you add a second VNIC interface (or a 3rd, 4th, etc), LInux will give it a name such as "ens5". To verify what the new label name is, download and run a script from Oracle.com. In the example below you can see the new "IFACE" is ens5.

Example commands

wget https://docs.cloud.oracle.com/iaas/Content/Resources/Assets/secondary_vnic_all_configure.sh

chmod a+x secondary_vnic_all_configure.sh

./secondary_vnic_all_configure.sh

Example of the script executing:

3) Assign the IP addresses to the network cards and update the MTU to 9000 (for Jumbo frames)

ip addr add 172.20.136.140/28 dev ens3 label ens3:0

ip link set ens5 mtu 9000

ip addr add 1.1.1.8/28 dev ens5 label ens5

ip addr add 1.1.1.10/28 dev ens5 label ens5:0

4) Create an Interface Configuration startup script for each VNIC and its floating IP addresses.

Your virtual router will have at least 4 interface startup scripts.

/etc/sysconfig/network-scripts/ifcfg-ens3 (primary-vnic)

/etc/sysconfig/network-scripts/ifcfg-ens3:0 (primary-vnic secondary IP)

/etc/sysconfig/network-scripts/ifcfg-ens5 (pod1-vnic)

/etc/sysconfig/network-scripts/ifcfg-ens5:0 (pod1-secondary IP)

The sample configuration for the startup script can be seen below. This will help ensure that the configuration is persistent upon reboots.

vi /etc/sysconfig/network-scripts/ifcfg-ens5
  DEVICE="ens5"
  BOOTPROTO=static
  IPADDR=1.1.1.8
  NETMASK=255.255.255.240
  ONBOOT=yes
  MTU=9000

5) Add static routes so that the Linux host knows how to reach your customer networks.

Ensure that VNIC routing entries are persistent upon reboots. To make the static routes persistent you can create a route- startup script. See the example below.

vi /etc/sysconfig/network-scripts/route-ens5
  172.20.138.0/24 via 1.1.1.1 dev ens5
  172.20.137.0/24 via 1.1.1.1 dev ens5

6) Enable IP Forwarding so your Linux host will start forwarding traffic between interfaces.

vi /etc/sysctl.d/98-ip-forward.conf
  net.ipv4.ip_forward=1

7) We need to enable "loose" reverse path forwarding to enable use cases where asymmetric routing could occur. For example, traffic comes in on vrouter1, but comes back on vrouter2, etc.

vi /etc/sysctl.d/97-reverse-path-forwarding.conf
  net.ipv4.conf.all.rp_filter=2

8) Restart the network:

systemctl restart network

If you need to add static routes for testing purposes, here are a few example commands.

ip route add 172.20.138.0/24 via 1.1.1.1 dev ens5
ip route add 172.20.137.0/24 via 1.1.1.1 dev ens5
ip route add 172.20.136.0/25 via 172.20.136.129 dev ens3