How do you keep Redis Sentinel failovers manageable under heavy traffic?

November 13, 2026 5 answers
Asked by two.spoons.short

We run Redis Sentinel on Kubernetes with one primary, two replicas and a Node.js backend using ioredis. After changing the termination hook to trigger failover and close connections, planned failovers take around 2 to 5 seconds in testing. The client queues commands until it reconnects.

What happens if a failover takes much longer under heavy traffic? I'm worried about the application running out of memory or overwhelming the new primary when the queue flushes. Is a managed Redis service likely to solve that?

Answers

_unclaimed_rubber_band

I would treat the offline queue as a buffer with a budget, not as a guarantee of availability. Estimate how many commands can arrive during a longer outage and how much memory they use, then put limits on incoming work before the process gets close to running out of memory.

Set request deadlines and decide what can fail fast. If Redis is just a cache for some endpoints, those might be able to bypass it or return a reduced response. Other operations may need to fail rather than wait indefinitely. An application timeout alone may not remove a command already queued by the client, so check that behavior when designing the limit.

Also test recovery, not just the outage. A large backlog can hit the promoted primary all at once, so limit concurrency or shed load while connections recover.

balcony-level-headed:

It is worth checking the timeout settings at the proxy and caller too. A command that finishes after the caller has given up can complicate retries, especially for writes.

Reply to this answer
margin-of-error-ish

Five to ten seconds of elevated latency during a failover may be an acceptable tradeoff, depending on the API. I wouldn't assume a managed service eliminates that gap. Compare its documented failover behavior and cost with your latency and availability targets before moving just for this reason.

Reply to this answer
somewhere_near_platform

Planned node drains and unplanned primary failures are different tests. Your termination hook can initiate a failover during a drain, but it won't run if a node disappears abruptly. Test both paths under representative traffic, including a longer outage and a slow recovery.

Measure queue growth, application memory, request latency and errors, plus how quickly the new primary handles the backlog. Use those results to choose a queue or concurrency limit and a deadline, rather than assuming the 2 to 5 second test result is an upper bound.

two.spoons.short:

That makes sense. The planned drain test doesn't tell us how much traffic we'd need to shed during an unexpected failure.

Reply to this answer
no_receipt_for_this

Be careful with the phrase zero downtime here. Queueing can hide a brief disconnect from callers who are willing to wait, but it doesn't make Redis available during the failover. Writes around a disconnect can also have an uncertain outcome, and asynchronous replication can lose writes that hadn't reached the promoted replica.

Review which operations are safe to retry and whether the application can tolerate that data loss window. If not, changing the client queue settings alone won't solve the problem.

Reply to this answer
shelves.with.legs

If the requirement really is continuous service through a primary failure, that's a bigger architecture question than tweaking Sentinel. Parallel access paths or lease-based coordination are possible designs to investigate, but they add complexity and don't automatically guarantee instant failover or safe writes.

I'd first define the maximum acceptable outage and data loss. A bounded pause may be much cheaper and simpler than designing for near-instant takeover.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions