When should I use Direct PUT to Firehose instead of Data Streams or MSK?
Asked by
MellowCedar42
I'm new to Kinesis and trying to understand when Amazon Data Firehose makes sense on its own versus putting Kinesis Data Streams or Amazon MSK in front of it. Firehose seems capable of handling high throughput and can deliver data directly to destinations such as Amazon S3, Redshift, or OpenSearch. When is it worth managing stream shards and paying for an additional streaming layer? Is the main reason to use Data Streams or MSK simply that Firehose's throughput limits have been exceeded, or are there other architectural reasons?
Answers
The choice isn't limited to situations where Firehose cannot handle the volume. A stream in front of Firehose is useful when producers and consumers need to be decoupled, when several applications need to read the same events independently, or when you need to reprocess older records. If none of those requirements apply and Firehose's supported sources and destination behavior are sufficient, Direct PUT is often the more straightforward option.
Reply to this answer
For collecting logs from many sources, especially on-premises systems, Fluent Bit can send records directly to Firehose and let it batch them into S3. In some cases that is simpler and cheaper than operating a separate streaming layer, particularly when the data only needs to be delivered to one final destination.
Reply to this answer
Direct PUT to Firehose is a good fit when you want buffered, mostly fire-and-forget delivery to a destination such as S3, Redshift, or OpenSearch, and the destination is effectively the only consumer. Firehose is a delivery pipeline rather than a durable event log, so throughput alone usually isn't the deciding factor. Use Data Streams or MSK when you need multiple independent consumers, replayable data with retention, or stronger control over ordering and stream processing.
Reply to this answer
Add your answer
Answers are reviewed before publication.
MellowCedar42: That clears it up—thanks. I was thinking about this mainly as a throughput question, but the replay and multiple-consumer differences are important.