I'm building a notification system for missed calls, alerts, and other messages, and I've run into the 1MB response limit of DynamoDB that everyone talks about. My users can accumulate thousands of notifications, and I need a way to let them mark all notifications as read instantly. Currently, when I query unread notifications, I end up retrieving way too much data, making it prohibitively expensive to update every single entry. I've tried using a timestamp in Redis to filter results but still end up paying for read units in DynamoDB for items that are technically marked as read. It feels like I'm constantly battling the database. Has anyone else implemented a notification feed like this? How did you tackle the "Mark All" feature? Did you use a watermark timestamp, or find some other clever way to batch update? I'd love to hear your experiences and tips!
5 Answers
Have you thought about adding a version field? Increment it when all notifications are marked read in a global user configuration. When retrieving notifications, you can just read the latest version, which simplifies the logic.
It sounds like your schema may need a little adjustment! Instead of marking each message as read, consider indexing notifications by a timestamp or a similar Sargable value. You could also use a watermark in the user record to manage the read status more effectively. This way, you can mark all messages as read in a single row update. If that's not feasible, setting up an SQS and Lambda pipeline to handle marking reads asynchronously could also work.
If you're okay with a bit of duplication and some associated costs, a simple approach would be to track message IDs that are unread for each user. This could be a separate record per user, and when they click "mark all as read," you can simply delete that record. It keeps the original notifications intact while allowing you to clear them all in one operation.
Another approach is to move the read/unread marker to the user record instead of attaching it to each notification. That way, the user record tracks the last read timestamp for notifications. So, when a user marks all notifications as read, you only need to update that timestamp. Then, you can easily check if there are new notifications added after that timestamp.
You might also consider creating a secondary index with the user ID and a 'hasBeenRead' flag. This can streamline the querying process and allow you to manage the notifications more efficiently.
Exactly! I worked at AWS for a while, and that's a solid strategy to avoid the overhead of updating each notification individually.