I'm moving my container workloads over from AWS ECS to EKS and I'm wondering if Kubernetes has a reliable circuit breaker feature similar to what ECS offers. In ECS, the circuit breaker stops deployments after multiple failed attempts. I've had issues with it not responding to internal container failures, and I'm curious if Kubernetes has a solution that works better. When I tried out the deployments, they seemed to go into a CrashLoopBackoff state instead of stopping properly. Any advice or insights on this?
4 Answers
Just a heads up, in Kubernetes, previous pods don’t terminate until new ones are up and passing liveness probes. If the update is problematic, your old pods keep running smoothly unless they fail a probe.
You're not alone in dealing with CrashLoopBackOff without a native circuit breaker in Kubernetes. One solid approach is running pre-deployment tests directly in your cluster. Tools like Testkube can help you run various tests as part of your deployment process, which can catch failures before you hit that CrashLoopBackOff state. It adds a layer of safety by catching issues early, but remember to still use liveness probes and canary deployments for extra protection. Happy testing!
Ah, interesting! We stick with Terraform as well for our deployments. Haven't dive into conventional methods yet, but we're looking at testing out ArgoCD or Flux soon to see if they suit our needs better.
It really depends on how you're deploying your apps. If you're using ArgoCD, for instance, you can leverage rollouts which resemble ECS's circuit breaker functionality. If you’re using Flux, Helm releases can help too. What deployment tools are you currently using?
That sounds like a solid plan! Exploring different tools can definitely help provide the reliability you’re looking for in Kubernetes.