I'm working on a small service that handles HTTP requests, background jobs, and possibly WebSocket connections. My current shutdown sequence is to mark the service unavailable, stop accepting new work, let active requests finish, flush a limited number of notifications, and then close the queue and database connections. The part I'm unsure about is how long to wait before forcing the process to exit. Which shutdown steps have proven valuable in production, and how do you choose a timeout without adding unnecessary complexity?
4 Answers
A short grace period such as 15 seconds is a reasonable starting point if your deployment can afford it. Stop routing traffic first, then let in-flight work finish. Watch logs and shutdown metrics, and increase the limit only for services that occasionally build up a meaningful in-memory event backlog. Different workloads may need different limits.
Make sure the service becomes unready before you stop accepting connections. Otherwise the load balancer or orchestrator may continue sending traffic until its next health-check update. Your application-level shutdown order can remain mostly the same, but readiness handling is often the detail that prevents new work from arriving during the drain period.
The grace period should account for the timeout used by the load balancer or proxy in front of the service. Give the system enough time to stop routing traffic and then allow active requests to finish—typically at least slightly longer than that upstream timeout. Beyond that, bounded cleanup is useful, but elaborate flushing and shutdown choreography usually matters less than reliable readiness changes and a firm forced-exit deadline.
Base the timeout on real measurements rather than guessing. Look at normal and worst-case request durations, backlog processing times, and any other shutdown work. A practical starting point is often several times the observed duration—for example, if completion usually takes under a second, try a timeout around 8–10 seconds—then enforce a hard limit so a stuck connection cannot keep the process alive forever.
That seems like a good baseline. I’ll measure actual request and backlog durations, then adjust the grace period based on the data.