How do you catch failures in things nobody thought to monitor?
Asked by
MellowPine42
We spend a lot of time tuning alerts and reducing false positives, but the more worrying problem is the opposite: systems failing without any monitoring at all. A few weeks ago, one of our queue consumers stopped processing overnight. That queue was an internal secondary component, so nobody had configured an alert for its backlog or inactivity. We only discovered it the next morning when downstream systems began failing, leaving several hours of work to process.
We already have a reasonable number of alarms, but it's impossible to anticipate every useful metric on every resource, especially as new services and queues are added. How are you handling these "unknown unknowns" without putting an alert on everything? Do baseline or anomaly-based systems reliably catch unplanned failures, or do they mostly create another source of false positives? I'm especially interested in practical approaches that catch silent failures while remaining maintainable.
Answers
Alert on proof that the system is producing useful results, not just on the health of every individual component. For a queue consumer, that might be output volume, downstream completion, or the age of the oldest unprocessed message. If the final result suddenly drops or becomes stale, you’ll catch a broken consumer without needing to predict every way it could fail.
Reply to this answer
Start with boundary and freshness checks: can users log in, are APIs responding, are dependencies reachable, and is the data being delivered within its expected age? For internal pipelines, “time since the last successful result” is often more meaningful than raw queue depth. Anomaly detection can help once behavior is stable, but bursty or seasonal workloads tend to make it noisy, so I’d use it after basic end-to-end checks and service-level objectives.
Reply to this answer
A dead-man’s switch is useful for workers that should always be running. Have each consumer send a heartbeat on a schedule, then alert when the heartbeat stops. It’s a simple way to catch a silent worker failure without tuning a threshold for every queue. I’d still pair it with a real-work signal, since a process can keep sending heartbeats while its useful processing path is broken.
Reply to this answer
Treat every incident as a coverage review. When something breaks in an unmonitored way, add the missing signal along with a documented response, automation, or runbook. You can’t prevent every new failure, but you can prevent the same blind spot from recurring. Also put this into the development process: every new queue, worker, or scheduled job should specify what it produces, how it proves it is alive, and how its failure will be detected.
Reply to this answer
Make monitoring part of the infrastructure modules instead of relying entirely on people to remember it. For example, a shared queue module could create an oldest-message-age or inactivity alert by default, with an explicit opt-out for unusual cases. A pull request for a new worker should also require an answer to “how would we know this stopped working within 30 minutes?” That prevents new blind spots while you gradually audit existing systems.
Reply to this answer
Add your answer
Answers are reviewed before publication.
MellowPine42: That makes sense. We’ve mostly been adding alarms after incidents, so putting the defaults and the coverage question into the service and infrastructure templates would probably help more than trying to discover every missing alarm manually.