I have two sites connected by a site-to-site IPsec VPN over gigabit links. Site A uses a FortiGate 400E running FortiOS 7.2, while Site B uses a FortiGate 120G running FortiOS 7.6. From Site B to Site A, iPerf3 reaches line rate with both TCP and UDP. In the opposite direction, UDP also reaches line rate, but TCP performs at less than 1% of the UDP speed. The VPN configurations are otherwise the same, no traffic profiles are affecting performance, and both VPN interfaces have TCP MSS set to 1418. The tunnel uses DH group 21 with AES256-GCM and PRF-SHA384. What could cause TCP to be slow in only one direction?
4 Answers
The FortiOS change to IPsec TCP MSS handling is worth checking, especially with different major versions on the two appliances. Even after upgrading both devices and confirming the clamp value, packet captures are still important because the configured value may not match the effective MSS on the actual traffic path.
Verify the effective MTU throughout the path and try lowering it temporarily. The interface MSS setting may apply to all traffic crossing that interface, whereas a policy-level clamp can target the VPN traffic more precisely. A mismatch between the FortiOS versions could explain why one direction behaves differently.
Capture both iPerf3 tests on each side and compare the TCP handshakes, retransmissions, packet sizes, advertised windows, and any ICMP fragmentation-needed messages. That should show whether packets are being dropped, fragmented, reset, or held up by windowing. Also test between the sites without traversing the VPN if possible, to separate a WAN issue from an IPsec issue.
This strongly suggests an MTU or TCP MSS problem. Recent FortiOS versions changed how MSS handling works, and automatic clamping may no longer behave the same way on both sides. The larger TCP packets may be getting dropped in one direction while the reverse direction is being clamped correctly. Check the TCP MSS value directly on the firewall policies, not just on the VPN interfaces.
Does setting TCP MSS on the interface take precedence over the policy-level MSS clamp?