I'm currently running a Kinesis stream to handle data flowing through my system, and I have Kinesis Data Firehose set up to send that data to S3. However, I'm concerned about the costs, as Firehose is costing me about twice as much as the Kinesis stream itself. Is this a normal scenario, or are there more cost-effective and reliable options to transfer data from Kinesis to S3? Also, just to clarify, I'm not performing any transformation, using a 128 MB buffer size with a 600-second buffer interval. We're dealing with a high volume of data, and it saves data in 128 MB files after 600 seconds.
2 Answers
That does seem a bit unusual! Are you also processing the Kinesis stream directly in your own system? That might be adding to the costs if you're duplicating functionalities.
It sounds odd to have Firehose alongside a Kinesis Data stream. Just be cautious of creating unnecessary complexities. Firehose can help with archiving data in a decoupled way, but it's worth exploring if you really need both.
I agree, but Firehose is indeed connected to the Kinesis stream for writing to S3, costing about $29 per TB.
Yes, Firehose is directly sourcing from the Kinesis data stream, which is costing around $29 per TB written to S3.