Maximum transmission unit and maximum segment size

Magic Transit has operation requirements that customers need to understand to ensure their network functions as intended. Pay particular attention to the maximum transmission unit (MTU) and maximum segment size (MSS) values. Incorrect configuration of these values can cause performance loss or prevent data packet ↗ delivery.

MTU and MSS

The maximum transmission unit (MTU) ↗ is a measurement representing the largest data packet that a network-connected device will accept. The MTU almost always applies to Layer 3 of the Open Systems Interconnection (OSI) model in networking and includes the entire packet, including all headers (Transmission Control Protocol (TCP), Internet Protocol (IP), etc.) and the data (payload) itself. For example, packets must not exceed 1,500 bytes to route through the Internet.

The maximum segment size (MSS) ↗ refers to the amount of data that you can send in a single TCP datagram packet. You determine this value by subtracting the size of the IP and TCP headers from the MTU, which instructs the router how large the payload can be. It applies to Layer 4 of the OSI model in networking.

One common misconception about MSS/MTU is that setting these values negatively impacts performance. While there is a slight performance penalty, it is worse not to configure these values to account for the specificities of your network.

Encapsulation

Since Magic Transit uses encapsulation to deliver its services, it is also important to understand why MTU and MSS matter in this case.

Encapsulation adds bytes to the packet because Cloudflare adds a new IP header and (often) some sort of encapsulating header to every packet. For example, in the case of Generic Routing Encapsulation (GRE) for Internet Protocol version 4 (IPv4), encapsulation adds 24 bytes — 20 bytes for the IPv4 header and 4 bytes for the GRE tunnel header.

A network interface that performs GRE encapsulation needs to account for the added overhead by reducing its MTU. Since the MTU maximum size is 1,500 bytes, for IPv4 this means the MTU becomes 1,476 bytes (the original 1,500 bytes minus the 24 bytes from the GRE encapsulation). This reduced MTU defines the maximum size of the IP packet that GRE can encapsulate.

Fragmentation

If the data packet is larger than what the network interface can accept, the network must either drop or fragment it into smaller packets. When fragmentation occurs, Cloudflare only accepts data packets that it can completely reassemble. If some fragments are missing, Cloudflare discards all received fragments. Cloudflare does not forward incomplete packets to the customer.

Setting the do not fragment (DF) bit in the TCP header to 1 denotes that the network must drop the packet rather than fragment it if the packet is larger than the MTU that intermediary network devices can accept. Most TCP implementations set the DF bit to 1 to avoid the potential issues that fragmentation causes.

If you experience issues with fragmentation and cannot set an MSS clamp, Cloudflare can clear the DF bit for you. When you enable this option, Cloudflare fragments packets greater than 1,500 bytes, and your infrastructure reassembles the packets after decapsulation. Use this as a last resort option. Contact your account team for more information.

Fragmentation in Magic Transit

Consider a UDP datagram of size 3,000 bytes (8 bytes for the UDP header + 2,992 bytes for the UDP data). To fit within a standard 1,500-byte MTU, this UDP datagram would be fragmented across three IP packets as follows:

Suppose that the UDP datagram has source port 389 and is destined for a Magic Transit customer IP address. Suppose also that the Magic Transit customer has a firewall rule in place that drops UDP traffic with source port 389, a common Connectionless Lightweight Directory Access Protocol (CLDAP) ↗ reflection attack vector.

The three packet fragments above will arrive at Cloudflare, but only the first fragment contains a UDP header with source port information. The second and third fragments contain UDP data but do not have UDP header information.

So the question is: which of these fragments does Cloudflare drop and which does it deliver to the customer? If Cloudflare only drops the first parts of fragmented packets, the remaining parts could still generate a large amount of traffic during a Denial of Service (DoS) attack.

How Cloudflare handles fragments

The following diagram shows how the three UDP fragments in the preceding example flow through Cloudflare and Magic Transit. The main takeaways are:

• Cloudflare never sends incomplete packets to customers: If Cloudflare does not see all parts of a packet required to fully reassemble that packet, Cloudflare will not send the partial data fragments to the customer.
• Magic Firewall operates on fully reassembled packets, not individual fragments: This means that filters that match on UDP/TCP header information, for example, apply to the fully reassembled packet, not just the initial fragment. Cloudflare will not leak non-initial fragments to customers.
• Customers can still see fragmented packets: By default (without clear_dont_fragment_bit set), Cloudflare fragments packets to fit within the configured MTU of the tunnel before sending the data to the customer. If a packet is larger than 1,476 bytes, Cloudflare will fragment it and send those fragments to the customer for reassembly.

In all cases, Cloudflare sends all fragments to the customer.

MSS clamping

Maximum segment size (MSS) is a TCP setting that limits the size of TCP segments. The SYN packets set this option during the three-way handshake.

By default, a TCP endpoint sets its MSS value based on its local network interface MTU. For example, for IPv4, if the MTU is 1,500 bytes then MSS becomes 1,460 bytes (1,500 bytes minus 20 bytes from the IPv4 header minus 20 bytes from the TCP header).

MSS is a tool that you can use to configure TCP packet size behavior. If a TCP endpoint sits behind a network with reduced MTU, changing the MSS value to match the actual path MTU value forces remote endpoints to send packets that fit within the specified MTU. So, if an IPv4 TCP endpoint sits behind a GRE tunnel with an MTU of 1,476 bytes, the MSS value in its TCP SYN packets should be 1,436 bytes - 1,476 bytes minus the 20 bytes from the IPv4 header, minus the 20 bytes from the TCP header.

One way to modify the MSS setting is by changing the MTU of the network interface in the router's WAN interface to match the path MTU. Another way to modify MSS is by applying an MSS clamp, where you configure an intermediary network device - such as a router - to modify the MSS TCP option on-the-fly when packets pass through it. Note that changing the MTU on the interface of an intermediary network device is not the same as applying an MSS clamp, and it does not change the TCP MSS value.

Refer to MSS clamping recommendations for information on what you should set your MSS clamping to, depending on the type of tunnel.

Warning

Cloudflare only recommends applying a MSS clamp to adjust the size of TCP packets. Changing the MTU of a network interface is not recommended as this might have unforeseen impacts on traffic.

MSS with Magic Transit and Direct Server Return

Asymmetric routing is a common scenario especially with Magic Transit. Ingress traffic from the Internet enters the Cloudflare network, then traverses a GRE tunnel (MTU of 1,476 bytes), and egress traffic from the datacenter is sent through Direct Server Return (DSR) over the Internet (MTU of 1,500 bytes).

In an asymmetric scenario, you need to reduce the MSS value of packets sent by Magic Transit users to the Internet to reduce the size of packets sent from the Internet towards their network. To accomplish this, you must configure either the customer's end-hosts or an MSS clamp on an intermediary device on the egress path of traffic leaving their network. The following chart details how MSS values affect payload sizes on both routing paths.

Key takeaway from the preceding chart: MSS clamping affects TCP packet payload sizes flowing in the opposite direction versus where the clamp is applied.

Tunnel-in-tunnel scenario with Magic Transit

MSS clamping only affects TCP traffic. If, for example, you have a web server on your Magic Transit prefix, then the MSS clamp takes effect on the TCP data from Direct Server Return (DSR) traffic. However, you need to take a different approach for any tunnels inside of your Magic Transit tunnel (tunnel-in-tunnel scenario).

For example, if you have a Magic Transit GRE tunnel set up, and then another IPsec or GRE tunnel running from third-party devices on your premises, MSS clamp has no impact on the outer packets of the encapsulated traffic. This is because MSS clamping affects only TCP traffic, and IPsec/GRE encapsulated traffic is IP. For this scenario, you need to lower the MTU of the internal tunnel interface further, both for your ingress and egress traffic.

MSS clamping recommendations

GRE tunnels as off-ramp

The MSS value depends on how your network is set up.

• Magic Transit ingress-only traffic (DSR):

  • On your edge router transit ports: Set a TCP MSS clamp to a maximum of 1,436 bytes.
  • On any IPsec/GRE tunnels with third parties on your Magic Transit prefix: Apply the MSS clamp on the internal tunnel interface (most likely on a separate firewall behind the GRE-terminating router) to reduce the current value by 24 bytes.

• For Magic Transit ingress + egress traffic:

  • On the Magic Transit GRE tunnel internal interface: Meaning where the Magic Transit egress traffic will traverse. Your devices may do this automatically once the tunnel is configured, but it depends on your devices. Set the TCP MSS clamp to 1,436 bytes maximum.
  • On any IPsec/GRE tunnels with third parties on your Magic Transit prefix: On the internal tunnel interface (most likely on a separate firewall behind the GRE-terminating router) to reduce its current value by 24 bytes.

IPsec tunnels

For IPsec tunnels, the value you need to specify depends on how your network is set up. The MSS clamping value is lower than for GRE tunnels because the physical interface sees IPsec-encrypted packets ↗, not TCP packets, and MSS clamping does not apply to those.

• Magic Transit ingress-only traffic (DSR):

  • On your edge router transit ports: Set the TCP MSS clamp to 1,436 bytes maximum.
  • On any IPsec/GRE tunnels with third parties on your Magic Transit prefix: On the internal tunnel interface (most likely on a separate firewall behind the GRE-terminating router) to reduce its current value by 140 bytes.

• Magic Transit ingress + egress traffic:

  • On your edge router: Apply this on your Magic Transit IPsec tunnel internal interface (that is, where the Magic Transit egress traffic will traverse). Your devices may do this automatically once the tunnel is configured, but it depends on your devices. Set the TCP MSS clamp to 1,360 bytes maximum.
  • On any IPsec/GRE tunnels with third parties on your Magic Transit prefix: On the internal tunnel interface (most likely on a separate firewall behind the IPsec-terminating device in your premises) to reduce its current value by 140 bytes.