I'm looking at event-driven options for a real-time Java system. How should I think about latency, scaling, and partition choices when comparing Kafka with other queues?
What are the hidden tradeoffs of event-driven design for real-time Java systems?
Asked by
7quarts_of_questions
Answers
Start by defining what fast means for the system. Kafka may be a reasonable fit for some workloads, but if low latency is the overriding goal, it's worth evaluating alternatives such as Chronicle Queue or a sequencer-based design from Coral Blocks.
Don't assume a queue will be faster just because someone recommends it. Compare them against the latency and throughput requirements of the actual workload, along with the operational and licensing costs.
Reply to this answer
Partition count is a scaling decision, not just a free capacity dial. More partitions can give consumers more opportunities to work in parallel, but that only helps if the event keys have enough variety to distribute the load.
Check the limits of the Kafka service you're planning to use too. For example, Amazon MSK can restrict cluster upgrades when there are too many partitions per broker. Adding brokers can address that limit, but reducing the broker count later may require recreating the cluster.
7quarts_of_questions:
So a higher partition count won't help much if most events end up on the same key?
sUbmit_when_ready_:
Right. A heavily used key can still concentrate work on one partition, even when the topic has plenty of partitions available.
Reply to this answer
Add your answer
Answers are reviewed before publication.
linen.and.logic:
Licensing and pricing are worth checking early. An available code repository doesn't necessarily make the terms of a commercial product clear.