How Can I Run Stable GPU Inference on AWS Without Spot Interruptions?

Could you explain how your current setup manages eviction notices? It sounds like you'd need a system that can migrate your tasks before those notices hit. You might explore AWS FIS to simulate evictions and check out Ray for Kubernetes (KubeRay). That framework has features like fault-tolerant task APIs, checkpointing for long-running tasks, and the ability to save progress when your setup gets a termination signal. You can also tweak things like `terminationGracePeriodSeconds` to give your pods some time to handle shutdown.

Here's my take: If you can't tolerate interruptions at all, then relying on spot instances may not be the best choice. They're not a one-size-fits-all solution.

Are you trying to reserve just one type of instance or multiple? If you request a variety of instances that can do the job, it could improve your chances. Just keep in mind with the AI boom, finding affordable GPU options is a real challenge!

If you're using SQS events, they should be able to manage interruptions before you get fully impacted. That's one strategy to check out!

Why not consider adding Kubernetes to your approach? That five-minute warning you get is usually sufficient to scale up and migrate workloads to a new node.

Honestly, I think there might not be an ideal solution. We've transitioned most of our GPU tasks to Lambda Labs since it feels like cloud prices are just skyrocketing for GPUs lately.