How does EKS gracefully drain pods during scale-in?
Asked by
MellowPine47
With AWS ECS, when a service scales in or deploys a new version, ECS deregisters the task from the load balancer and drains existing connections before sending SIGTERM to the container. This gives in-flight requests time to finish and helps avoid random 5xx responses.
We may need to migrate to Amazon EKS, so I'm trying to understand how Kubernetes handles pod termination during scaling or deployments. From what I've read, Kubernetes may terminate the pod while the load balancer is removing it at roughly the same time. Many examples seem to rely on a lifecycle preStop hook that simply sleeps for a while, but that still appears to allow occasional failures for requests that are in flight.
Is graceful shutdown really this unreliable in EKS, or are there recommended Kubernetes and AWS load-balancer settings—such as readiness gates, endpoint removal, connection draining, or termination grace periods—that provide behavior comparable to ECS?
Answers
Kubernetes is designed to handle this through pod termination and endpoint removal rather than immediately killing a pod. When termination begins, the pod is marked as not ready and removed from the service’s endpoints, so new traffic should stop being routed there. Existing requests can then finish during the termination grace period before the container receives its final termination signal.
You still need to configure a reasonable terminationGracePeriodSeconds value and make sure the application handles SIGTERM properly. A preStop delay can be useful in some setups, but it should be treated as a coordination buffer—not the entire shutdown strategy.
Reply to this answer
The Kubernetes pod termination documentation describes the sequence: termination starts, the pod is removed from service endpoints, and the containers are given time to shut down before they are forcefully killed. EndpointSlices are an important part of this process.
In practice, make sure your readiness probe becomes false during shutdown, your application stops accepting new work after SIGTERM, and the grace period is long enough for normal requests to complete. No system can guarantee that an indefinitely slow or already-broken request will finish, so the timeout still needs a hard limit.
Reply to this answer
For EKS, the details depend on whether you use the AWS Load Balancer Controller, an ingress controller, or another integration. Check the AWS load-balancing guidance for readiness gates, target registration mode, deregistration delay, and connection draining.
With the right target type and readiness-gate configuration, the load balancer can wait until the pod is considered ready and can drain it during termination. A short preStop sleep is sometimes recommended to allow propagation through the load-balancing layers, but it should be combined with proper readiness and deregistration settings rather than used as a workaround by itself.
Reply to this answer
Add your answer
Answers are reviewed before publication.
QuietHarbor62: The general draining pattern existed in load balancers and lifecycle hooks long before containers. Kubernetes uses the same idea, but the exact timing depends on the ingress or load-balancer integration and how quickly it observes endpoint changes.