I manage more than 10 separate containers with Docker Compose, but I'm not permitted to modify the existing docker-compose.yml files to add labels, sidecars, or other settings. I need a host-level solution that watches all running containers and automatically restarts any container whose health check reports unhealthy, without manual intervention. Would a daemon, Docker event listener, systemd service, or scheduled script be appropriate? If I write my own watcher using docker events or docker ps, what practices help prevent duplicate restarts, race conditions, and restart loops?
4 Answers
Before building a polling loop, check whether an external Compose override file is allowed. A docker-compose.override.yml can add labels, health checks, restart behavior, or related settings without changing the original file. This keeps the deployment changes separate and is generally easier to maintain than a custom script. It will not help if the restriction forbids all Compose configuration, but it is worth clarifying.
A host-level watcher can monitor the Docker socket and restart containers when their health status changes. An auto-healing service or a small Python daemon can do this, and it can be configured to target every unhealthy container rather than relying on labels. Use event notifications where possible instead of constantly polling docker ps, track containers currently being handled, and add a cooldown or maximum retry count so one failing service does not trigger repeated restart loops. Be careful with Docker socket access because anyone who can control it effectively has root-level control of the host.
Restarting an unhealthy container is recovery, not guaranteed 24/7 availability. If the container is the only instance, there will be an outage while it restarts, and a restart may not fix an application, dependency, storage, or configuration problem. For real availability, run redundant replicas behind a load balancer and make sure the application can tolerate failures. A workload orchestrator such as K3s, Kubernetes, or Nomad is usually a better fit once the environment needs scheduling, rescheduling, and health-based replacement.
If the deployment files truly cannot be changed, an external watchdog is still possible, but it cannot provide seamless failover by itself. It only automates the same restart a person would perform.
If you implement the watcher with systemd, make it a single long-running service rather than multiple cron jobs. Subscribe to Docker health-status events, verify the container still exists and is unhealthy immediately before restarting it, and serialize actions per container. Add rate limits, logging, backoff, and a failure threshold. Also distinguish a health-check failure from a stopped container, because restarting on every transient health result can make an already unstable service worse.
A generic auto-healer may still need labels or other container configuration depending on how it selects containers. A custom host-side watcher avoids that requirement, but it still needs careful handling of stale events and containers that disappear during a restart.