I'm designing a simple graceful-shutdown sequence for a small Node.js service that handles HTTP requests, background jobs, and possibly WebSocket connections. The plan is to mark the service unready and stop accepting new work, allow active requests to finish, flush bounded notifications, then close the queue and database connections. The uncertain part is deciding how long to wait before forcing the process to exit. What shutdown steps have proven valuable in production, and how should I choose a practical timeout without adding unnecessary complexity?
4 Answers
The core sequence is already sensible: stop advertising readiness, stop accepting new work, drain active work, apply bounded flushing, and then close shared resources. The parts most likely to cause trouble are forgetting the readiness transition and allowing shutdown handlers to wait indefinitely. Notification flushing and connection cleanup should have short, explicit limits; anything that cannot finish within the overall deadline should be abandoned safely.
Coordinate the grace period with the load balancer, proxy, or orchestrator in front of the service. The service should have enough time to become unavailable and drain requests before the platform forcibly kills it—often at least a little longer than the upstream routing or request timeout. Also make readiness fail before closing listeners, otherwise health-check and routing delays can continue sending new requests during shutdown.
A fixed grace period is a fine starting point. Around 15 seconds works for many services, especially when traffic is distributed across multiple instances and clients can be routed elsewhere. Check shutdown logs and telemetry afterward, then increase the limit only for workloads that genuinely need more time, such as draining an in-memory event backlog. Don’t let rare cases dictate an oversized timeout for every process.
Base the timeout on actual service behavior rather than picking a completely arbitrary number. Measure normal and worst-case request or job completion times, then leave some margin—often several times the observed duration. If requests usually finish in under a second, a timeout around 10–20 seconds might be reasonable, but make sure it also covers known background-job and WebSocket behavior. Anything that exceeds the deadline should be cancelled or terminated so shutdown cannot hang forever.
That makes sense. I’ll make readiness fail first, measure real request and backlog durations, and set the forced-exit deadline with the upstream timeout and drain delay in mind.