What self-hosted queue would you use for FIFO processing per key?

November 16, 2026 5 answers
Asked by q.813!buttonledger

I'm looking for a cloud-agnostic queue for a Node and TypeScript app on Kubernetes. Jobs need to run in FIFO order with only one in flight per key, while different keys can run in parallel.

Traffic is normally around 20,000 to 30,000 jobs an hour, but occasional bursts can reach 2.5 million in an hour. Is Kafka worth the complexity, or is there a simpler option?

Answers

q.604+amberwrench

Kafka is a reasonable fit if you can live with partitions being the unit of ordering. Send the same key to the same partition, and keep processing serial within that partition. Different partitions can be consumed in parallel, but a consumer group cannot use more active consumers than it has partitions.

The catch is that unrelated keys can share a partition. A slow job for one key can hold up the others behind it, so this is stricter than the per-key concurrency you asked for. Partition count and how keys are distributed matter a lot, and running Kafka is more infrastructure than a simple worker queue.

_e37!parcelspanner.9:

Also make sure retries don't let a later job for the same key complete first. Ordering in the log alone doesn't guarantee ordered completion if the consumer processes records concurrently.

Reply to this answer
8u!canvasratchet_426

I'd look at BullMQ with Redis for a Node app before taking on Kafka. It may be a simpler operational fit, but check the exact grouping and concurrency features available in the version and edition you're considering.

If you build the grouping yourself, you need more than a lock per key. Jobs also have to be selected in the right order, and a failed or retried job must not let later jobs for that key jump ahead. Separate queues per key are one possible design, though that means managing those queues too.

Reply to this answer
6a+buttonratchet_391

Pulsar's Key_Shared subscription is another option to investigate. It routes messages with the same key to one consumer at a time, while different keys can be handled by different consumers. You still need to process each key serially in the consumer and check how retries and redelivery affect your ordering requirements.

The tradeoff is operations. Pulsar and BookKeeper on Kubernetes are not a tiny setup, so I'd compare that cost against whether avoiding Kafka's partition-level bottleneck actually matters for your workload.

Reply to this answer
_e72!signalcrank.5

Before picking a broker, find out how many distinct keys are active during that 2.5 million job burst and how long jobs take. With one job at a time per key, a rough upper bound is active keys divided by average seconds per job. A hot key can still build a long backlog even if other workers are idle.

The queue holds the backlog while consumers catch up. It doesn't make the work disappear. Size storage and retention for the burst and decide how long you're willing to wait for it to drain.

Zorvani.4m:

That estimate assumes the active keys can all stay busy. Uneven key traffic and slower jobs can make the actual throughput lower.

Reply to this answer
Navorik-3e

For a burst that large, look at message size as well as message count. One option is to store payloads separately and put only their IDs in the queue. That reduces the data held by the broker, though the IDs and queue metadata still need space.

It adds a dependency on the payload store, so consider what happens if a worker can't retrieve an item or the queue message and stored payload get out of sync.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions