Hey folks! I'm currently working on deploying a microservice within an EKS cluster and need some advice on the right storage solution. Given that I'm bound by certain organizational constraints, I'm planning to deploy this service across multiple nodes—each node might host 1-2 pods depending on traffic.
Here's the scoop:
- The service makes about 500 API calls and performs data transformations before writing the final output to a storage layer.
Initially, I thought about using EBS for its performance, but since it doesn't support ReadWriteMany, it won't work for concurrent access from multiple pods on different nodes. I've also looked into DynamoDB and MongoDB, but they raise some concerns regarding cost and latency. In-memory storage is off the table since I need data persistence.
Right now, I'm leaning towards using Amazon S3 because it allows shared access among the pods, is cost-effective, and has enough latency tolerance for what I need. However, I'm trying to figure out how to avoid duplicate writes to S3 when multiple pods are writing simultaneously. Ensuring idempotency is my main priority.
Once I get the data to S3 reliably, I want to hook up a Grafana Agent to visualize the metrics from the bucket. If anyone has suggestions or experiences to share, especially around avoiding duplicates and using Grafana with S3, that would be super helpful! Thanks!
1 Answer
You might want to consider using EFS; however, S3 is a solid choice for your needs. It's affordable, durable, and offers great data management options. One approach to avoid duplicate writes is to store write keys in a DynamoDB table and check for existence before writing to S3. That may help manage conflicts across your pods.
That's a neat idea! But do you know how to manage concurrent writes to S3 or EFS? I’d love more info on that.