I'm currently working with a client who relies heavily on EC2 spot instances for their ECS clusters to keep costs down. Recently, I've noticed that our high-load applications (about 100 HTTP requests per second) aren't being drained from the target group quickly enough when the spot instances are about to terminate, which leads to annoying HTTP 502 Bad Gateway errors from the Application Load Balancer. We already have an agent that listens for termination notices to tell the target group to start draining the affected host.
I've read that AWS also sends out "EC2 Instance Rebalance Recommendations," which is supposed to be a warning that your instance might be reclaimed soon due to high demand. I tried subscribing to these events through EventBridge, but in the few I've analyzed, the recommendation often arrives either just before or at the same time as the termination notice.
Has anyone else been through this? How do the rebalance recommendations relate to the termination notices? And are there other AWS solutions to work around this, aside from switching to on-demand or reserved instances? We're trying to avoid that due to budget constraints.
1 Answer
It might help to tweak your application's request timeout and the target group's de-registration delay. The default delay is 300 seconds, which is longer than the 120 seconds AWS gives for a spot termination notice. Check the longest running requests and adjust your application timeouts accordingly. Just remember, the rebalance recommendations could come before the termination notices, giving you a heads up to manage interruptions proactively!
We've definitely run into those timing issues too. We found out our de-registration delay was set to just 30 seconds, and we bumped it to 70 seconds. This way, with a keep-alive time of 60 seconds, we can drain the app safely before termination.