We're working on a deployment that consumes messages from AWS SQS, and we want to implement a circuit breaker pattern. The goal is to pause message consumption when there's an issue with a downstream system. Since our deployment doesn't serve HTTP requests, we don't need a readiness probe like usual. One of my coworkers suggested using a readiness probe to check the health of the downstream system and use Kubernetes' Ready/NotReady status as a way to manage our circuit breaker. While I think this approach could work, I feel like it's a bit off-label or misapplied. Am I being too picky here? The alternative we're considering is to store the circuit status in Redis and check it before fetching messages from SQS, which could ensure that if one pod has its circuit open, all of them do since we need Redis anyway. What do you all think?
3 Answers
Your coworker’s suggestion has some merit if the main goal is to prevent the pod from receiving traffic, which is what the readiness probe is meant for. However, since you're pulling work from a queue instead, I think using Redis would be cleaner and more efficient. You don’t gain much by using readiness since it’s not really designed for your specific use case.
Consider using a readiness gate instead. It would allow an external observer to control the gate status that determines your pod's readiness. Check out Kubernetes docs on readiness gates for more info. But I have to ask, attaching a gate to readiness makes me wonder—are you sure it suits your needs? Since you’re focusing on a downstream system's health, is there a benefit to storing that state on the pod instead of Redis?
Have you looked into Keda? It allows you to scale workloads to/from zero based on various metrics. It could help manage your circuit breaker situation more smoothly. But I’m curious, can you stack scalers in a single deployment? Like, could you scale up based on SQS message volume but then scale back to zero if the circuit status dictates it?
Related Questions
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically
[Centos] Delete All Files And Folders That Contain a String