Since last Friday, our VPN tunnels have been intermittently dropping when connecting to locations that use AT&T, while tunnels over providers such as Comcast remain stable. We tested the same site over both services and could reproduce the problem only on the AT&T connection. The issue began suddenly around 8 AM Eastern, continued through late Monday, temporarily disappeared, and has now returned. Our locations are in Central and South Florida.
AT&T says there are no problems with the routers, network, or recent firmware changes. We have not changed the configurations on the AT&T gateways—mostly BGW320s, with some CGW450-400 units—or on our firewalls. For troubleshooting, we tried IP passthrough and disabled the gateway firewalls. ActiveArmor is also disabled on the gateways we checked.
Has anyone seen similar behavior recently? Could this be an AT&T peering or routing problem, a gateway state-table issue, or something else outside our firewalls and VPN configuration?
4 Answers
This sounds like it could be an AT&T interconnection or peering problem rather than a local VPN configuration issue. There have been reports of unstable paths around some Southeast interconnection points. Since your affected sites are in Florida, compare the route and packet loss to the VPN peer over both providers, especially while the tunnel is failing.
The gateway may still maintain connection and NAT state even when it is in passthrough mode. Older gateway hardware had relatively limited state tables, and a saturated or corrupted table could cause intermittent tunnel failures. Rebooting the gateway can temporarily clear it; if that consistently helps, compare behavior after a reboot and consider testing with the gateway bypassed through a supported direct handoff if AT&T allows it.
ActiveArmor has caused connectivity problems for some BGW320 installations, so it is worth checking, but disabling it does not completely rule out a gateway-related issue. Make sure it is disabled on every affected gateway and that no security profile or service is still active.
When one of the tunnels drops, test the remote peer independently through the Comcast circuit. If the peer responds normally over Comcast while the AT&T path fails, collect timestamps, traceroutes, packet-loss results, and the public source addresses for AT&T escalation. That evidence should help distinguish a gateway problem from a route or peering flap.
That lines up with the timing and the fact that Comcast stays stable. We’ll capture route and loss data from both circuits during the next failure.