I'm new to Kinesis and trying to understand when it makes sense to send records directly to Kinesis Data Firehose versus putting Kinesis Data Streams or Amazon MSK in front of it. Firehose seems to handle high-throughput delivery on its own, so I'm wondering why I would manage stream shards and take on additional cost if Firehose can write directly to destinations such as Amazon S3, Redshift, or OpenSearch. Is the main reason to use Data Streams or MSK simply that Firehose's throughput limits have been exceeded, or are there other architectural differences I should consider?
3 Answers
Direct PUT can also be a cost-effective way to batch logs from many unrelated sources into S3. For example, an on-premises logging setup using Fluent Bit can send records straight to Firehose without requiring you to provision and manage stream shards. It’s a good fit when the data mainly needs buffering, batching, and delivery rather than real-time fan-out or repeated processing.
Direct PUT is usually the right choice when you need a buffered, managed delivery path into a destination like S3, Redshift, or OpenSearch, and the destination is effectively the only consumer. Firehose is a delivery service rather than a durable event log, so the decision usually isn’t about raw throughput. Use Data Streams or MSK when you need multiple independent consumers, replayable data with configurable retention, or stronger control over ordering and stream processing. If you only need to collect events and land them in a sink, adding another streaming layer may just increase cost and operational work.
Think of Firehose as a managed ingestion-and-delivery pipeline. It accepts data from supported producers and forwards it to configured destinations, but it isn’t intended to expose the same event stream to several applications or let you replay old records. Choose Data Streams or MSK when applications need to consume the events directly, when several consumers need their own offsets, or when you need to retain and reprocess the original stream before delivery to a final destination.
So the practical question is less about which producers Firehose supports and more about whether the events need to remain available as a stream for other applications before or after they reach the destination.
That clears it up—so the key distinction is replay and independent consumers, not simply whether Firehose can handle the traffic.