I'm new to Kinesis and trying to understand when Amazon Data Firehose makes sense on its own versus putting Kinesis Data Streams or Amazon MSK in front of it. Firehose seems capable of handling high throughput and can deliver data directly to destinations such as Amazon S3, Redshift, or OpenSearch. When is it worth managing stream shards and paying for an additional streaming layer? Is the main reason to use Data Streams or MSK simply that Firehose's throughput limits have been exceeded, or are there other architectural reasons?
2 Answers
For collecting logs from many sources, especially on-premises systems, Fluent Bit can send records directly to Firehose and let it batch them into S3. In some cases that is simpler and cheaper than operating a separate streaming layer, particularly when the data only needs to be delivered to one final destination.
The choice isn't limited to situations where Firehose cannot handle the volume. A stream in front of Firehose is useful when producers and consumers need to be decoupled, when several applications need to read the same events independently, or when you need to reprocess older records. If none of those requirements apply and Firehose's supported sources and destination behavior are sufficient, Direct PUT is often the more straightforward option.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures