I'm looking for a sanity check from other UK sysadmins. Since Monday, we've had intermittent connectivity failures affecting some sites and several home-user connections, mainly in London. Our hosted firewall sits outside the sites and handles ingress to file servers and VPN access. Connections to it sometimes fail completely for ten minutes or more than an hour: traceroute stops responding, and pings to the site gateways show 100% packet loss. The same problem can occur in the opposite direction, from the hosted firewall toward the site gateways. The firewall provider appears to have valid BGP routes and works normally for many other sites and ISPs, so I'm not yet convinced it's the cause. We're mainly seeing this with Hyperoptic Business, Community Fibre, Vodafone, and some CityFibre-connected services. Has anyone else noticed regional or ISP-specific routing problems recently?
5 Answers
Can you identify the upstream or last-mile provider for each affected circuit and check what the relevant prefixes look like from outside your network? Also, when you say traceroute doesn’t resolve, do you mean the destination itself fails to resolve in DNS, or that the trace stops receiving replies? If it’s DNS resolution, check which DNS servers are authoritative and compare results using several public resolvers.
I haven’t seen a widespread issue myself, so this may be regional or specific to particular last-mile providers. For critical sites, it would be worth having two genuinely separate carriers and termination points rather than relying on one access network.
We have several sites across northern England using Vodafone circuits and haven’t noticed any interruptions. That makes a broad Vodafone outage seem less likely, though a regional routing problem could still be affecting specific exchanges or paths.
I’m also seeing normal service on a CityFibre connection in the North East, as well as at an office using another provider. The location and exact transit path may be more important than the ISP name alone.
Run an MTR from one affected site and one unaffected site while the problem is happening. If both paths start failing or showing loss at the same hop, you’ll have much stronger evidence to give the ISP than simply reporting that the connection drops. Capture results in both directions if possible.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures