I experienced a huge gap in our monitoring setup during yesterday's Cloudflare outage and I'm curious if others faced similar issues. Despite a flood of alerts indicating that systems were down—everything seemed fine on our end, from CPU usage to logs and health checks. It turned out there was a Bot Management bug on Cloudflare's side, which caused us to think our origin services were completely down. This led us into a series of futile troubleshooting steps, like restarting services and rolling back changes, which was a complete waste of time.
The real concern is that none of our monitoring tools could effectively differentiate between a failure on our end and an issue with the CDN or edge. Everything just showed up as 'DOWN' with no context. I've been trying to work on a solution that can identify when the CDN is down but the origin is still functioning, or vice versa. Has anyone built a system for this, or found tools that can help tell these differences, particularly with services like Cloudflare, Akamai, CloudFront, or Vercel?
4 Answers
I totally get your frustration. Sometimes just checking the Cloudflare status page can clarify a lot quickly. Our monitoring setup also fires alerts before checking there, which adds more confusion. There’s definitely a need for more sophistication in how alerts are triggered based on the source of the issue.
It's crucial to have monitoring checks for DNS resolution, server connectivity, and CDN functionality correctly set up. Make sure to alert on DNS issues separately from origin problems. A minimal setup should include checks from both inside and outside your environment to catch these failures distinctively.
That makes sense! We've implemented some of this, but it still felt like an origin failure during the outage. Could you share how you manage to keep those checks clean between CDN and origin?
We built a custom program that taps into Cloudflare's status page API along with others we use. It helps us internalize those statuses to make useful monitoring checks. Having an endpoint that checks the current status of services gives us quick insights during outages.
That’s a very clever approach! Might explore implementing a similar method in our system.
Layering these checks seems effective—especially if you can independently verify whether it's a CDN or origin issue based on those metrics.
You’ve hit a common failure point. A good practice is to split your checks by path. Maintain one check that hits the origin directly and another through the CDN. Tagging alerts helps distinguish between 'origin dead' and 'edge issues.' Using separate DNS and TLS checks can also provide a clearer view of what’s actually failing instead of merging everything into a single alarm.
This sounds like the right direction. Our incident yesterday made it clear we need that separation to avoid confusion in alerts. Thanks for the detailed strat!
That's true. Cloudflare's diagnostics can help a lot if you catch it early. We need to adjust our monitoring to ensure it first checks the third-party services before panicking!