I'm exploring options for streaming data in AWS and need some guidance. I have payloads that are under 1 KB in size and I'm looking to handle about 100 of these per second. The goal is to stream these payloads in real-time to a data store for another service to read them efficiently. Additionally, I want to ensure that there's an option for permanent storage as well. I initially considered AWS Elasticache (Redis), but I found it to be quite costly and it doesn't support permanent storage. What services would you recommend for my use case?
5 Answers
DynamoDB could also work well, especially if you want to retain handled payloads for up to a year. It handles the load you're describing well, though costs could add up with high read throughput. If maintaining order is essential, make sure to implement that in your design.
If you want an open-source option, consider using MSK (Managed Streaming for Kafka) with Firehose to S3. It's an effective yet budget-friendly solution for your streaming and storage needs.
Given your requirements, AWS Kinesis is a solid starting point for streaming data in real-time. For permanent storage, S3 is typically the go-to choice. You might want to consider using Kinesis Data Firehose to stream your data into S3, allowing for both real-time processing and durable storage.
Thanks! Just to clarify, will Kinesis provide permanent storage?
For your case, you could use a combination of Kinesis for real-time streaming alongside Firehose to write that data into S3 for permanent storage. This setup is efficient and takes care of your streaming and storage needs without being overly complicated.
You might also want to look at using SQS for queuing unprocessed items, especially if you're worried about maintaining order while your consumer reads the data. Additionally, if your need for permanence implies retaining handled data for a certain time, that’s something to factor into your architecture planning.
I appreciate the suggestions! It's really helpful.
Yes, keeping the order is important since I'm building a stock price streaming service.