When you provision a compute instance in OCI, it automatically comes with jumbo frames enabled by default, configuring the network interface with a Maximum Transmission Unit (MTU) size of 9000 bytes to enhance throughput and reduce protocol overhead.

However, a key consideration is how OCI manages traffic destined for the internet, where the standard MTU is only 1500 bytes. The solution lies with the Internet Gateway. When packets are sent to the internet via the Internet Gateway, they are limited to a maximum size of 1500 bytes. Any packets exceeding this MTU will be dropped. Therefore, the source instance must adjust its MTU for traffic heading to the internet to avoid packet loss.

Traffic coming into OCI, however, is handled differently. If traffic is coming from the internet, it cannot exceed an MTU of 1500 bytes. However, if the packet was fragmented at the source or at an intermediate hop, the fragments are accepted, and the destination system is responsible for reassembling the packets.

Purpose of this blog is the discuss how the OCI will handle incoming fragments (udp fragments) at following levels

  • Network security lists/Groups
  • Compute instance
  • Network load balancer

Network Security List/Groups 

When it comes to UDP fragmented packets only the first packet has the UDP header information rest of the packets will not. Check out a packet capture of a sip packet.

First packet with header and subsequent packets with no header information.

sip-packet-with-header

When fragmented packets reach a security list or security group, they are dropped by default if the security policy permits only specific port traffic, since the UDP header is missing. To address this, it’s recommended to add a security rule that allows traffic on all ports between the specified source and destination, as this traffic cannot be restricted by port.

For TCP packets, Oracle advises using Path MTU Discovery (PMTUD). To support proper MTU negotiation, OCI security lists allow ICMP packet type 3, code 4 by default.

Compute Instance

OCI compute instances, as previously mentioned, have a default MTU of 9000. Once traffic is permitted by the security list or security group, the instance can receive these fragments and successfully reconstruct the packets. Additionally, there are no limitations regarding fragmentation at the VNIC level.

OCI Network Load balancer

The OCI Network Load Balancer (NLB) cannot process fragmented packets, meaning any UDP packets that arrive in fragments will be dropped. This poses a challenge for customers, as they often cannot remove the NLB from their architecture due to redundancy or load balancing needs.

While there are several ways to address this issue, this blog will focus on a specific workaround.

The solution involves redirecting incoming traffic to a third-party device that can reconstruct the fragmented packets and then forward them to the NLB. Although the NLB can handle packets with a higher MTU, it struggles specifically with fragmented packets. In this blog, I will demonstrate how to use a pfSense firewall to perform packet reconstruction. Many next-generation firewalls have similar capabilities, allowing them to reassemble fragmented packets for inspection before sending them to their final destination.

Note: If your application will be receiving huge number of fragmented packets make sure the firewall in the path has enough compute and memory to handle traffic.

For step-by-step instruction on how to deploy a PFSense firewall follow this blog.

This is the test enviroment to show how end to end traffic will work.

udp-test-env

 

The idea here is to transparently route traffic through pfSense. We’ll enable jumbo frames on the LAN interface so that when a fragmented packet arrives at the WAN interface, the firewall will reassemble it and forward it out through the LAN interface without further fragmentation. Since jumbo frames are supported at the NLB level, if the NLB receives an unfragmented packet, it can forward it to the backend without issue.

Now, let’s take a closer look at the incoming packet on the firewall’s WAN interface.

pf_wan_interface

 

On the WAN interface we are receiving one packet with three fragments. Since on the LAN interface we have jumbo frames enabled the packet that is reassembled by the firewall is forwarded as is rather than getting it fragments again.

pf_lan_interface

 

Let’s look at the packet received at the backend through the NLB.

test_instance_backend

Same packet is received on the backend system as well through the NLB. So NLB was successfully able to forward this packet.

In conclusion, compute instances in OCI can successfully handle fragmented packets if security policies permit, but the Network Load Balancer (NLB) presents a challenge, as it cannot process fragmented UDP packets. A potential solution to this problem involves using a pfSense firewall to reconstruct fragmented packets before forwarding them to the NLB. This method allows the NLB to receive unfragmented packets, ensuring proper delivery to backend systems.