I noticed this morning that no email had arrived since around 5 AM Pacific, so I checked MX Guard Dog's website and found a major outage notice. Their primary Phoenix data center suffered a complete cooling-system failure, causing the facility to lose connectivity. The provider has been working on restoring cooling, but there is still no firm recovery estimate. MX Guard Dog recommends that customers who can do so temporarily change their MX records to point directly to their own mail servers. Is anyone else affected, and what steps are you taking to keep mail flowing?
4 Answers
Phoenix being hot probably made the situation harder, but the larger issue is the cooling failure and lack of a second operating site. A properly resilient facility should be able to handle an HVAC failure without taking the entire mail-filtering service offline.
The outage highlights a major weakness: having multiple MX records does not help much if they all lead back to the same physical data center. A mail-filtering provider should have geographic and infrastructure redundancy, not just multiple servers in one location. Most senders will queue messages and retry, so mail is likely delayed rather than permanently lost.
If you bypass the gateway, first confirm that your mail server accepts connections from the entire internet. Some organizations only allow inbound mail from the filtering provider’s IP ranges. You should also increase your own spam and phishing protections, since the gateway’s filtering will be gone, and watch SPF and DMARC reports for legitimate mail that used to pass through the provider.
Also lower the MX-record TTL before switching back, and keep both paths available for a while afterward. DNS caching means some senders will continue using the old destination even after the record changes.
We had to redirect several domains directly to our mail servers. The change may take time because of DNS caching, but most legitimate systems should keep retrying delivery. Once service is restored, I’d seriously consider moving to a provider with multiple regions and independently redundant data centers.
They had advertised fault tolerance and multiple MX records, which is why a lot of customers assumed there was real failover. This outage makes it clear that the redundancy was not geographically independent.