We're working on a deployment that processes messages from AWS SQS and want to set up a circuit breaker pattern. The idea is that when there's an issue with a downstream system, we can pause message consumption. Since our deployment doesn't handle HTTP traffic, a typical readiness probe seems unnecessary. One of my coworkers suggested using a readiness probe to monitor the health of the downstream service and leverage the Ready/NotReady state through Kubernetes API calls from within the same pod as a way to indicate circuit status. While that might function, I feel like that could be misusing the readiness concept. Am I being too critical, or do others see a concern here? Another option we've considered is using Redis to manage the circuit status, which would be beneficial since it can maintain a shared state across all pods in the deployment. We're already using Redis, so there's no additional infrastructure to manage.
3 Answers
Using a readiness probe could make sense if your intention was to prevent the pods from receiving traffic altogether. But considering your main task is to pull from a queue, using readiness doesn’t really add any real benefit. I think going with Redis would be a cleaner and simpler solution, honestly.
Have you looked into Keda? It can help you scale workloads to zero based on various conditions, including custom scalers. This could be a neat way to manage your consumption without overcomplicating things.
That sounds interesting! But can I combine multiple scalers in a single deployment? I want to ensure my deployment can scale up with SQS messages but also scale down to zero if the circuit is open.
A readiness gate could actually be a good fit for your situation. It allows an external source to control the on/off state. You might want to check out the Kubernetes documentation for more on this!
But what's the actual advantage of attaching this to readiness? The core concern is about the health of the downstream system, so why not store that status in Redis instead? It seems more straightforward.
Exactly! The purpose of the circuit breaker is to stop all pods from pulling messages when there's an issue.