What are some practical ways to catch quiet failures when there isn't already an alert for the affected component? I'm thinking of things like an internal queue consumer stopping without anyone noticing until downstream work fails.
Does baseline or anomaly detection help with this, or does it just create more noisy alerts?
Start with the outcome that queue is supposed to produce. Is the downstream result still arriving and reasonably current? An alert on output freshness or an unexpectedly low amount of completed work can catch failures across several internal components, including ones you didn't think to monitor individually.
For the queue itself, oldest message age can be more useful than depth when volume is low. A small queue can still contain work that's been stuck for hours.
Use a missing heartbeat alert for processes that are supposed to run continuously. Instead of waiting for the worker to report an error, alert when its expected signal stops arriving.
Just be clear about what the signal proves. A process can be alive and still fail to do useful work, so a signal tied to successful processing is stronger than one that only says the process is running.
Test the paths that matter from the outside too. Can users log in? Do key endpoints respond, DNS resolve, and certificates remain valid? Those checks won't tell you every root cause, but they can expose a problem before someone has worked out which internal metric was missing.
Anomaly detection is worth considering, but I wouldn't assume it solves the blind spot automatically. Bursty or seasonal workloads can make a normal pattern look suspicious, and you'll still need to decide which signals deserve attention and what someone should do when they fire.
Try it on a few useful signals and check whether the alerts lead to action before relying on it more broadly. An alert with no clear response can become another thing people learn to ignore.
Some gaps will get found the hard way. When an incident exposes one, include the missing detection in the follow-up work, along with how someone should respond. That won't catch every new failure, but it helps avoid being surprised by the same one twice.
Answers are reviewed before publication.
Nivlet.9c:
Freshness needs an expectation that fits the workload. If work doesn't arrive on a regular schedule, a quiet period alone may not mean anything is broken.