This is the 3rd blog, (Part 3B), in a series of blogs regarding the ISV Validated Design.
This blog series will contain the following topics
In this blog we will focus on the OCI Instances and how to configure them to support the large scale design for ISV's. In the last blog entry we focused on the networks, route tables, and security lists needed. Now we will focus on how to setup the virtual router instances.
Think about the quantity of PODs, and the total bandwidth you need between your Management networks and your End Customer instances. Choose an Instance shape that is appropriate for the bandwidth you'll need, the quantity of VNICs, etc.
By default your POD network can peer with up to 10 Customers networks. With a service limit request, you can increase that limit from 10 to 20.
Each customer should have their own Dynamic routing gateway or Internet Gateway, so in theory most of the customer traffic should go out the DRG, or Internet Gateway.
Most traffic should be "control plane" traffic (from Customer instances to Management Instances), versus "data plane" traffic (i.e. Customer 1 accessing their application running in the Customer 1 VCN over the Internet/VPN, etc).
You can change the Instance shape if needed, but it might require stopping the instance.
Virtual Router Instance details
Screenshot - Creating a new compute instance. Make sure to give your router a name!
Screenshot - If you click on "Change Image Source, and then click on the "Oracle Images" section you can deploy the Oracle Autonomous Linux instance.
Screenshot - Pick an Availability Domain, Instance Type and Shape
Screenshot - Choose a shape. Review the Network Bandwidth (aggregate across all interfaces) and the Total VNICs (or PODs you can attach to). For example if you select the 2.16 shape, you'll have 1 primary VNIC, and you can attach up to 15 PODs.
Screenshot - Attach the primary VNIC to the vrouters subnet
Screenshot - Choose a Fault Domain. Make sure to place subsequent routers in different Fault Domains.
In the HA Configuration section of this document we will go into more detail about how to move the Floating IP Addresses from one host to another.
Breadcrumbs: Compute >> Instances >> Instance Details >> Attached VNICs >> VNIC Details >> IP Addresses
This section was validated and tested on Oracle Linux 7.x, and should work on most RedHat/CentOS based distributions.
To setup your Linux instance as a dual homed router, I would recommend taking the following steps:
1) Log into the host via SSH.
2) Verify the network interface label for the new VNIC
When you add a second VNIC interface (or a 3rd, 4th, etc), LInux will give it a name such as "ens5". To verify what the new label name is, download and run a script from Oracle.com. In the example below you can see the new "IFACE" is ens5.
chmod a+x secondary_vnic_all_configure.sh
Example of the script executing:
3) Assign the IP addresses to the network cards and update the MTU to 9000 (for Jumbo frames)
ip addr add 172.20.136.140/28 dev ens3 label ens3:0
ip link set ens5 mtu 9000
ip addr add 126.96.36.199/28 dev ens5 label ens5
ip addr add 188.8.131.52/28 dev ens5 label ens5:0
4) Create an Interface Configuration startup script for each VNIC and its floating IP addresses.
Your virtual router will have at least 4 interface startup scripts.
/etc/sysconfig/network-scripts/ifcfg-ens3:0 (primary-vnic secondary IP)
/etc/sysconfig/network-scripts/ifcfg-ens5:0 (pod1-secondary IP)
The sample configuration for the startup script can be seen below. This will help ensure that the configuration is persistent upon reboots.
vi /etc/sysconfig/network-scripts/ifcfg-ens5 DEVICE="ens5" BOOTPROTO=static IPADDR=184.108.40.206 NETMASK=255.255.255.240 ONBOOT=yes MTU=9000
5) Add static routes so that the Linux host knows how to reach your customer networks.
Ensure that VNIC routing entries are persistent upon reboots. To make the static routes persistent you can create a route-
vi /etc/sysconfig/network-scripts/route-ens5 172.20.138.0/24 via 220.127.116.11 dev ens5 172.20.137.0/24 via 18.104.22.168 dev ens5
6) Enable IP Forwarding so your Linux host will start forwarding traffic between interfaces.
vi /etc/sysctl.d/98-ip-forward.conf net.ipv4.ip_forward=1
7) We need to enable "loose" reverse path forwarding to enable use cases where asymmetric routing could occur. For example, traffic comes in on vrouter1, but comes back on vrouter2, etc.
vi /etc/sysctl.d/97-reverse-path-forwarding.conf net.ipv4.conf.all.rp_filter=2
8) Restart the network:
systemctl restart network
If you need to add static routes for testing purposes, here are a few example commands.
ip route add 172.20.138.0/24 via 22.214.171.124 dev ens5 ip route add 172.20.137.0/24 via 126.96.36.199 dev ens5 ip route add 172.20.136.0/25 via 172.20.136.129 dev ens3