Hey everyone! Hope you're all doing well! I'm working on a low-latency architecture that needs to operate in **sa-east-1** and handle a significant amount of data. I plan to store customer lists for **access control**—if a customer is on a list, they can proceed on a specific journey. There will be **N journeys**, each with a separate list.
I'm considering using an **S3 bucket** where I split the data into files using a **deterministic algorithm**. This approach should help me know exactly where each customer ID is stored, allowing me to load specific files into memory with my Lambda function and minimize reads from S3. Each file would contain roughly **100,000 records (IDs)** and nothing more. I'm targeting around **20ms latency** using **AWS Lambda and API Gateway** (company requirements). Do you think this setup is feasible, or should I explore other options?
4 Answers
Fetching data from S3, along with parsing, could push you over the 20ms mark. I suggest moving to DynamoDB and using JOURNEY ID as the partition key and the user ID as the sort key to achieve O(1) lookups within single-digit milliseconds.
Cold starts with Lambda could be a significant issue if you’re aiming for a 20ms target. You’d need to clarify your requirements—like if that’s a strict SLA or a median expectation. Provisioned concurrency can help reduce cold start times, and depending on your traffic, it might make sense to think about Fargate.
I'm curious how you would set up the primary and sort keys in DynamoDB to avoid hitting partitions while still achieving quick lookups.
S3 isn’t really designed for low latency; it’s more for storage. You might face issues getting under that 20ms response time with it, especially given data fetch times.
I think you should really consider using DynamoDB for your storage. What you’ve described could turn into a complex and buggy solution if you rely too much on S3. DynamoDB is specifically made for quick lookups and should fit your needs better.
Great point! Scattering the partition key can help, but it might add to your latency concerns.