Why Does Changing MTU on Physical Hosts Affect Kubernetes Communication?

Just a quick note: MTU discovery is generally more effective with TCP than with UDP or IPSEC, and ICMP packets can often get blocked. It's best to keep a standard Ethernet MTU of 1500 to avoid these complications. Ensure that all your switches support this as well. If they're all in the same VLAN, having matching MTUs on all host interfaces should prevent most communication issues.

MTU discovery can be tricky! It relies on different network layers to process ICMP messages correctly, but in Kubernetes, that can fall apart fast. When you increased the host MTU, Calico and your overlay probably started sending larger packets, but if there’s any device in the path that can't handle it, you encounter issues. With UDP, if packets are lost, the app doesn't retry, leading to what feels like random lag instead of clear failures. The default 1450 MTU in Calico is typically safe because it accounts for encapsulation overhead. If you want to raise it, you need every network component — including switches and NICs — to support the same MTU, otherwise you risk running into this kind of rabbit hole.

If you're suspecting MTU problems, consider conducting an 'MTU ping test.' It helps establish the optimal MTU size on your router and can provide some insights into your issues.