Introduction
IPsec tunnel mode is widely used for site-to-site VPNs, especially on firewalls and enterprise routers. While the VPN may appear to work without issues, improper MTU configuration often causes hidden performance problems such as slow transfers, broken applications, or intermittent connectivity.
This article explains what IPsec tunnel mode is, how it affects MTU, and why MTU and MSS tuning is critical for a stable VPN.
What Is IPsec Tunnel Mode?
IPsec tunnel mode encrypts the entire original IP packet, including the source and destination IP addresses. A new IP header is added to route the encrypted packet across the network.
[ New IP Header ][ Encrypted Original IP Packet ]
Tunnel mode is commonly used when:
- Building a site-to-site IPsec VPN
- No GRE or overlay tunnel is used
- Internal IP addresses must be hidden from the public network
Most firewall-based VPNs use tunnel mode by default.
IPsec Tunnel Mode Overhead
Tunnel mode adds multiple headers to each packet:
- New IP header
- ESP header and trailer
- Authentication data
This increases packet size by approximately 40–60 bytes, depending on encryption and hashing algorithms.
Because of this added overhead, packets that fit normally in a 1500-byte Ethernet frame may exceed the path MTU once encrypted.
Why MTU Problems Occur in IPsec VPNs
Most Internet and ISP networks support a maximum MTU of 1500 bytes. When IPsec tunnel mode adds overhead, the resulting packet can exceed this limit.
This leads to:
- Packet fragmentation
- Packet drops
- TCP retransmissions
- Slow or stalled applications
The VPN tunnel may stay “up,” but traffic performance becomes unreliable.
Why Fragmentation Is Dangerous in IPsec
Fragmentation in IPsec is particularly problematic because:
- Fragmentation often occurs after encryption
- Many devices drop encrypted fragments
- NAT devices handle fragmented IPsec traffic poorly
- Path MTU Discovery (PMTUD) depends on ICMP, which is often blocked
As a result, MTU issues can be difficult to diagnose and reproduce.
Recommended MTU and TCP MSS Configuration
To avoid fragmentation, it is best to reduce the tunnel MTU and adjust TCP MSS.
Example Cisco IOS configuration:
interface Tunnel0
ip mtu 1400
ip tcp adjust-mss 1360
Explanation:
- Lowering the MTU ensures encrypted packets fit within the underlay network
- TCP MSS clamping prevents hosts from sending oversized TCP segments
This configuration works reliably across Internet, MPLS, and mixed networks.
Why Jumbo MTU Is Not a Solution
Increasing tunnel MTU to values like 9000 does not solve the problem because:
- Tunnel MTU does not change the physical path MTU
- Most Internet paths do not support jumbo frames
- Large packets increase fragmentation and packet loss
Jumbo MTU is suitable only in fully controlled private networks, such as data center fabrics.
Best Practices for IPsec Tunnel Mode
- Use tunnel mode only when GRE or overlays are not required
- Always tune MTU and TCP MSS
- Avoid relying on fragmentation or PMTUD
- Keep MTU values consistent across VPN peers
Conclusion
IPsec tunnel mode is secure and reliable when properly configured, but MTU tuning is essential. Without adjusting MTU and TCP MSS, VPNs may suffer from poor performance and unstable connections even though the tunnel appears operational.
Correct MTU configuration is not optional — it is a core requirement for stable IPsec VPN deployments.
Troubleshooting command
the check the right MTU and MSS we can discovery with this ping in linux/unix
ping -g 1568 -G 1480 -h1 -D google.com 
