Building a Scalable OCI Network Architecture for IoT

Internet of Things (IoT) is one of the latest technology revolutions that allows devices of almost any shape and size to communicate with each other and other systems in real-time. To handle the volume and velocity of the data generated by IoT devices, it’s important to build a scalable and robust network architecture to support these connectivity requirements. OCI provides a robust network platform for creating such an architecture. Here’s a guide to building a scalable, resilient, and secure network architecture for IoT on OCI. 

Basic Concepts

Many components are required for this communication to work successfully, but at a high level, we will make use of the following Publisher-Subscriber model:

• A standard protocol: The MQTT protocol will fit the bill here. It’s understood by many IoT clients and is a lightweight publish-subscribe that’s straightforward to configure.
• A Publisher: This is the IoT client that will send messages to anyone subscribed to receive them.
• A Subscriber: This is IoT is client configured to receive messages from a specific publisher. Taken a step further, this client subscribes to a particular “topic” that the publisher has listed for sending messages. 
• MQTT Broker: A central location that receives messages from publishers, processes them, and then forwards them to any subscribers of a topic. This is key to scalability as it decouples publishers and subscribers from needing to be directly connected.

Putting all these components together will give you an overall design similar to the following:

Understanding Requirements

Before diving into the architecture, let’s outline the key requirements:

1. The architecture should have the ability to scale, handling an increasing number of devices and messages without performance degradation.
2. Ensure high availability and fault tolerance to maintain continuous operations between IoT publishers and subscribers.
3. Protection of data in transit by ensuring only authorized devices can communicate with each other.
4. High performance by guaranteeing real-time or near-real-time data transmission between publishers and subscribers.

Network Architecture Overview

OCI can easily handle all of the key requirements listed above. Before diving deeper into the network components let’s look at the overall design example of what can be deployed using a single OCI region:

In this scenario, external IoT devices will be considered publishers and will push messages to a cluster of MQTT brokers located behind an OCI Public Load Balancer (LB) in a central VCN. This load balancer also has the Web Application Firewall service attached to it as well. Allowing it to inspect MQTT traffic between the Publishers and MQTT brokers.

When one of MQTT broker processes the message, it will duplicate the message to other MQTT brokers in the cluster, this is indicated in the diagram by the dashed red line between the brokers. This ensures syncronization between the IoT subscribers and brokers, as each subscriber will make a persistant connection with a specific broker. The MQTT brokers will each forward the message to their intended subscribers located in spoke VCNs via the Dynamic Routing Gateway.

Separating workloads (Prod, Dev, Test, etc.) per VCN allows for easier security management, access, and control. This level of segmentation also allows additional capabilities to easily be added in the future as new features and functionality is added to the IoT service. This includes storing data in a scalable Autonomous DB, streaming the data to other OCI analytics services, or even taking advantage of our GenAI services!

Network Components

The network architecture is made up of different network components divided between infrastructure to host resources and the network appliances to support and secure the resources:

Infrastructure

• VCNs: The VCNs provide the framework for provisioning network components and resources. A separate VCN for Dev allows for safely testing performance, patches, and new features on IoT Subscribers before rolling them out to your Prod VCN.
• Subnets: Public subnets will deployed for the external-facing public load balancer in the hub VCN to receive all incoming traffic from IoT Publishers.
• Security Lists and Network Security Groups: Provides the first layer of network security by limiting access to specific sources and destinations based on IP address, TCP/UDP port, and protocol. To increase network performance make use of stateless rules. IoT communication consists of many short-lived flows between Publishers and Subscribers that can quickly consume space in the connection tracking table of resources in OCI.
• Internet Gateways (IGW): The entry point for Publisher traffic into the OCI infrastructure. The IGW will only be provisioned in the hub VCN and only the public LB will route traffic in and out of this gateway.
• Service Gateways (SGW): These gateways are optional and will be used to connect to any additional OCI SaaS services your IoT architecture will need to make use of. Such as Logging Analytics or Functions
• Dynamic Routing Gateway (DRG): The DRG is a virtual router that provides connectivity between different private networks. This includes on-prem, other OCI regions, or even other cloud providers.

Network Appliances/Services

• Load Balancer: Located in the Hub Public Subnet, the LB will be assigned a public IP and will intercept any incoming messages from external IoT Publishers and distribute them to its backends.
• LB Backends:  While not a network component, the MQTT Brokers will be added as backends to the Public Load Balancer.
• Web Application Firewall: This OCI service includes feature-rich security features such as OWASP compliance-specific rules, access controls based on geolocation, and bot management. The WAF policy is attached to the public load balancer
• OCI Network Firewall (Alternative): Our NFW provides deep packet inspection for data passing in and out of your OCI infrastructure. Consider this as an add-on or alternative to using the WAF, which is primarily only concerned with layer 7 web traffic. More thoughts around this in the below central VCN section.

Central VCN Design

Let’s take a closer look at the network components deployed specifically in the central/hub VCN. As this will be the focal point for all IoT traffic to enter and exit the OCI environment.

The first stop after passing the Internet Gateway (IGW) is the OCI Load Balancer. The Load Balancer is highly available between Availability Domains or Fault Domains within an OCI Region. It’s also flexible and can scale bandwidth throughput along with auto-scaling MQTT brokers as necessary to support increased throughput.

Note that the LB also has an OCI Web Application Firewall policy attached to it. This will allow the LB to properly inspect incoming packets before passing to the MQTT brokers. It’s recommended to secure MQTT traffic with the use of TLS encryption (TCP port 8883). With TLS encryption it’s recommended to decrypt this traffic at the LB with the use of SSL/TLS certificates. This will allow the WAF to properly inspect the entire payload before passing it on to an MQTT broker.

An alternative to a WAF is our OCI Network Firewall (NFW) product, this opens up additional capabilities such as Intrusion Detection and Protection. In this scenario, the Hub VCN design changes slightly:

*Note if TLS encryption is used it’s recommended to also grant the NFW with the capability to decrypt this traffic with SSL/TLS certs for the same reason as the WAF.

Both design approaches should be measured in detail to ensure it meets all four requirements listed at the beginning of this guide. The WAF and NFW can be paired together but this could potentially limit the high-performance requirement for real-time communications. As each MQTT packet will hit both security checkpoints, get inspected, and possibly be re-encrypted in the path between the IoT publisher and subscriber. An analogy would be going through both Customs and the TSA when landing at an airport. While very important for security it does incur additional delays before moving on to the final destination. 

Conclusion

Once the architecture is built it’s important to perform validation testing to ensure that what designed meets all requirements and works as expected. I’ve created the example scenario in this blog and confirmed that I was able to communicate with Subscribers hosted in OCI from an IoT Publisher located externally from OCI. Below is a snippet of my testing from my Mac acting as an IoT Publisher:

I used an open-source MQTT Python Client to test from the IoT Publisher:

I also took advantage of other open-source MQTT services such as Mosquitto for my MQTT brokers.

• More info about this product can be found here.
• We also have a blog covering the installation process.
• I cover more details about monitoring, inspecting, and securing MQTT data in OCI in the following blog.

Building a scalable and resilient network architecture for IoT involves careful planning and the use of various services. By leveraging many of OCI’s cloud-native services, this can be accomplished efficiently and straightforwardly. Allowing your team to focus on IoT innovation, delivering value to your customers and spending less time managing the underlying infrastructure.