I'm currently using DynamoDB for my project, but I've hit a wall with its throughput limits. Specifically, DynamoDB restricts writes to 1000 per second per partition, and some of my customers need balance updates that exceed this limit. MemoryDB looks like a persistent version of Redis and I'm wondering if it would be a good option for handling high-volume balance updates.
2 Answers
You might want to consider deepening your hash key structure, like using something like 'partner:subaccount'. This adjustment can help you utilize more partitions. But are you actually trying to hammer updates to a single row? That could be a bottleneck.
There are strategies like bucket hashed counters in DynamoDB where you can divide one counter into multiple buckets. You can then use a QUERY operation to fetch all the buckets (with the partition key as the counter ID and sort key as the bucket value) and sum them up for the total count. By using hashing for incrementing, you can spread the load across different buckets, which definitely helps with scaling. There are a few articles out there that discuss various methods for this!
Unfortunately, that might not work well due to the 1k TPS limit being per partition. In DynamoDB, the smallest space you can partition is by a single primary key. This means all the sort keys under that primary key reside in the same partition under maximum load. If you’re looking to scale, you’d need to shard the primary key and make separate GETs for each shard. Some articles cover this sharding technique if you want to dive deeper.
Exactly! To really scale with DynamoDB under heavy load, sharding the primary key is essential. It’ll allow you to distribute the writes better, but it requires additional get operations for each shard, which can be a bit of extra work.