I'm designing a DynamoDB schema where each school can have millions of students. For example, if I use `SCHOOL#{id}` as the partition key and `STUDENT#id` as the sort key, all the students from one school will end up in a single partition, potentially causing hot partitions. I'm considering sharding the key (like `SCHOOL#{id}#SHARD#{n}`) to distribute the load better. How can I determine the right number of shards? What sharding strategy works best in DynamoDB? I'll be needing to paginate student data for the school admin with operations like ListStudentsBySchoolID, AddStudentByID, and so on.
5 Answers
Hot partitions are less of a concern now with DynamoDB's adaptive capacity. It'll manage partitions based on how you access your data, so if you're starting fresh with a well-thought-out schema, you should be fine. However, if you're planning to load data in bulk, keep in mind the scaling might behave differently. For partition counts, think about how many concurrent queries you'd need.
Good news is that DynamoDB now automatically handles partition splits even on sort keys. If usage patterns are balanced, this shouldn't be an issue as long as you're not targeting a small subset too aggressively—like hitting the same student ID with thousands of requests. Also, try to avoid monotonically increasing sort keys like timestamps to keep access more random.
Thanks, that gives me more confidence in using DynamoDB!
Sharding your data could complicate things. While it helps spread load, it might break some access patterns. If your access isn't too heavy on a particular school, you might be okay using just the school ID with UUIDs or similar methods for added uniqueness in the partition key.
That’s helpful! I think I'll stick with a simpler schema for now.
To avoid hot partitions, think about your access patterns first. It’s crucial to know how you’ll access your data so you can properly design your key schema. Sometimes DynamoDB might not be the best option if your data access won't be evenly distributed.
I've updated my question with the access patterns I'm considering.
Consider using a different partition key like student last names or user IDs. If you anticipate heavy request loads on a single school, diversifying the partition key might help. However, if you have many schools, composing the partition key with both school ID and student ID could be effective and less cumbersome than sharding.
I have many tenants so I think including both IDs would be wise. Thanks!
I see. I’ll need to plan around my querying needs!