I'm running a complex setup using Docker Compose for a distributed workflow that goes from Redis to a backend through a bridge and workers. Initially, I relied on basic instance heartbeats to identify dead containers, which helped with crashes but didn't indicate if services were operational, like whether Redis was reachable or if there were dependency timeouts. So, I created a three-layer health check system consisting of liveness checks (for Docker restart policy), readiness checks (to examine dependencies), and per-container instance heartbeats.
Furthermore, I implemented a separate `watchdog-services` container that periodically checks the readiness of each service and updates a global circuit breaker flag in the database if there's a degradation. This approach clarifies failure modes, such as system degradation when the engine or Redis is down, and allows for easier debugging during outages. For those managing production systems with Docker Compose, how do you manage health checks and service dependency failures? Is your logic distributed among services, or do you have a centralized approach?
2 Answers
I wouldn’t say it’s overengineering at all; it’s essentially what Kubernetes offers, just in a more explicit manner. In production with Compose, I’ve used both distributed health checks (with each service monitoring dependencies and exposing `readyz`) and a centralized approach like your watchdog. The latter really shines when you're looking for system-wide behavior modifications, like a 'degraded' state. Just be aware of the complexity that comes with it because if the watchdog falters or has a problem writing to the database, it creates another layer of failure. I usually keep local checks for liveness, make readiness dependency-aware, and use the backend to manage higher-level degradation decisions and not rely on an extra service.
That sounds reasonable! Balancing simplicity and coordination really is the key. Keeping it manageable while you learn is always a smart strategy.
The three-part split you've got with liveness, readiness, and heartbeat checks is definitely the right way to go. I've run a similar setup, and I found that not letting Docker’s restart policy handle everything is crucial. Just a heads up, be cautious with that external watchdog; if it’s relying on the database to flip the circuit breaker and the DB goes down, you'll face a single point of failure there. I solved this by writing the watchdog state to a shared tmpfs volume to avoid issues if the DB itself fails. Also, ensure to set a shorter timeout for your `readyz` checks than the healthcheck interval. If Redis is running slowly, a prolonged readiness check can mislead Docker into thinking the checker is unhealthy.
That's a great observation on the watchdog being a potential SPOF. In my setup, the circuit breaker flag does live in the DB, which leaves us blind if the DB fails. Writing to a shared volume seems like a smart move! Do you use that as a source of truth, or just as a last-known status? And I'm totally with you on the short timeouts for Redis—those slow responses can definitely lead to confusion!
The SPOF issue with the watchdog is something we faced too, but we tackled it with an AI agent that monitors the health checks and can analyze failures across services instead of just randomly restarting them. The key is making sure the watchdog understands the dependency structure, not just polling endpoints.
Totally fair—rebuilding some orchestration features makes sense. I designed my system from the ground up to fully explore all potential failure models before jumping into using an orchestrator. Right now, I manage liveness and restarts locally, keep readiness focused on dependencies, and let the watchdog just signal degradation without affecting the primary control flow. And yes, I'm deliberately using Compose for the time being; it keeps things straightforward as my system size is still small, but I know that might change as complexity increases.