A nightly backup recently stopped running—or began failing—and nobody noticed because the host stayed healthy, uptime checks remained green, and no alert or ticket was generated. The problem was discovered only when a restore was needed.
How do you detect the "job went quiet" situation rather than just a server outage? Do you use a hosted dead-man switch such as Healthchecks or Cronitor, a self-hosted tool like Uptime Kuma, Prometheus with a Pushgateway, a custom heartbeat script, or simply email from the job itself? How do you handle grace periods, success-only heartbeats versus explicit failure signals, and notification channels such as email, chat, or paging? I'm especially interested in practical approaches for a small team monitoring only a handful of important scheduled jobs without building a large observability system.
5 Answers
Emailing a success message can be enough for a very small environment: put messages into daily or weekly folders and notice when one is missing. It doesn’t scale particularly well, though—people miss one message among many, schedules change, and a successful command does not prove the output is useful. For critical backups, the stronger practice is to verify the result by periodically performing a test restore, not merely checking that the cron process returned zero.
Prometheus works if you already have it. Important jobs push a success metric and timestamp after completing, then alerts check both that a successful run occurred within the expected window and that the timestamp is still advancing. For backups, it’s worth exposing or collecting backup-tool metrics as well, since a job can technically succeed while producing an unusable or incomplete backup.
Passive monitoring with a TTL is another solid option. Each job reports completion to Icinga, Nagios, or a similar system and extends its deadline—for example, a daily job might receive a 25-hour TTL. If the next report does not arrive, the monitoring system raises the alert. This avoids the circular problem of creating a second cron job to check the first one.
The standard pattern is a dead-man switch: the job sends a heartbeat only after completing successfully, and an external monitor alerts when that heartbeat becomes overdue. The heartbeat can be a file with an expected modification time, an HTTP request, or a metric pushed to a monitoring system. The important part is that the alerting mechanism is independent enough that a broken scheduler or host cannot also take down the checker.
For a small setup, Uptime Kuma’s push monitors work well. Put a heartbeat call at the end of the wrapper script, and configure the monitor with a grace period slightly longer than the expected schedule. If the job never runs or exits before the heartbeat, Kuma eventually alerts. A hosted service such as Healthchecks provides the same basic model and can also handle start, finish, and milestone signals without much setup.
That’s the pattern I was leaning toward too. The main question for us is whether to use a local result-file check or push the heartbeat somewhere external so we don’t have to rely on another host-local mechanism.