I manage more than 10 containers across several Docker Compose projects. We are not permitted to edit the existing docker-compose.yml files to add labels, sidecars, or other container-level configuration. I still need a host-level solution that monitors container health continuously and automatically runs docker restart when a container becomes unhealthy. What external watchdogs or daemons can monitor all containers, and what is the safest way to implement this with systemd, docker events, or periodic polling without race conditions? I also want to understand whether restarting an unhealthy container is enough for real availability, or whether a more resilient architecture is needed.
4 Answers
A host-level service can watch the Docker socket and restart containers whose health status changes to unhealthy. You can use an auto-healing tool configured to target all containers, or write a small service in Python or another language that listens to Docker events. Be careful: access to the Docker socket is effectively root-level access, so protect the host service and avoid exposing the socket unnecessarily.
If you build the watchdog yourself, prefer Docker event notifications over a tight cron loop. Keep a per-container lock or in-memory state so repeated health events do not trigger multiple restarts. Before restarting, query the current inspection result again, confirm the container is still unhealthy, and record a cooldown timestamp. Also handle containers that are removed or recreated while the check is running, and log every action.
A periodic systemd service can still work, but it should be idempotent: inspect the container immediately before acting, use a timeout, and allow only one restart operation per container at a time. Otherwise a slow restart can overlap with the next polling cycle.
Before adding automation, verify that every container actually has a meaningful Docker health check. An unhealthy status only exists when a health check is configured, and restarting can hide the underlying problem if the check is failing because of a dependency, bad configuration, or resource exhaustion. If changing the original Compose file is forbidden, check whether a docker-compose.override.yml is allowed; it can add deployment settings without modifying the original file.
Restarting an unhealthy container is recovery, not guaranteed 24/7 uptime. If the only instance is stopped or restarting, that service is unavailable. For genuine availability, use multiple replicas behind a load balancer and make sure the application can recover its state. A workload orchestrator such as K3s, Kubernetes, or Nomad can provide rescheduling and health-based replacement, though that may be more infrastructure than this setup needs.
That approach fits the restriction, although some auto-heal tools normally depend on labels in Compose. I would need one that can select all existing containers or a host script that does the filtering itself.