I'm looking for some guidance on streaming small data payloads into an AWS data store in real time. My incoming payloads are under 1 KB in size, and I have about 100 payloads coming in every second. Additionally, I need the capability for permanent storage so that another service can access these payloads later. I've considered AWS Elasticache (Redis) but found it to be too costly and unsuitable for permanent storage. What AWS services would you recommend for this setup?
5 Answers
For a more budget-friendly setup, consider using SQS plus Lambda to write messages to S3 periodically. It's a bit hands-on to get running, but it's cost-effective and can serve your streaming needs quite well.
You mentioned needing "permanent storage"—it's important to clarify if that means indefinite retention or just retention of unprocessed items. With your needs, DynamoDB could handle your throughput, but costs might add up, especially at 100 reads per second. Make sure to check if order matters for your payloads as that could influence your choice.
SQS or MSK (Managed Streaming for Kafka) could also be good options, depending on your needs. If you're looking for more permanent storage, make sure to check if MSK will allow you to access the data after a week.
You might want to start with AWS Kinesis for streaming your data. It's designed for real-time processing, and many setups often use S3 for durable storage afterwards. Kinesis streams can keep data for a maximum of 365 days, so you have some flexibility there.
If you're focused on maintaining your data in JSON format for querying later, Kinesis Data Firehose streaming directly into S3 might be your best bet. It's straightforward and serverless!
Related Questions
How to Build a Custom GPT Journalist That Posts Directly to WordPress
Cloudflare Origin SSL Certificate Setup Guide
How To Effectively Monetize A Site With Ads