I'm looking for a cloud-agnostic queue for a Node and TypeScript app on Kubernetes. Jobs need to run in FIFO order with only one in flight per key, while different keys can run in parallel.
Traffic is normally around 20,000 to 30,000 jobs an hour, but occasional bursts can reach 2.5 million in an hour. Is Kafka worth the complexity, or is there a simpler option?
_e37!parcelspanner.9:
Also make sure retries don't let a later job for the same key complete first. Ordering in the log alone doesn't guarantee ordered completion if the consumer processes records concurrently.