I'm working with an AWS Lambda that processes data records triggered by an event. For simplicity, let's say it takes in one record and sends out ten records to a Kinesis stream. If something goes wrong during the output (for example, a service interruption while writing records), how can I ensure that I recover without losing or duplicating any records?
In a scenario where 9 out of 10 records are successfully written but the Lambda fails afterwards, the same input would be fed to the function again on trigger re-invocation. This would mean duplicating the output records, since the function would try to create all ten records again. My initial thought is to use a deduplication method involving a unique hash stored in a DynamoDB table. This would enable checking if a record has already been processed. Is this the best approach, or are there other strategies I could consider?
2 Answers
One option is to just let duplicates happen and handle them as soon as possible. For example, you might use a 'last write wins' approach in your data store to resolve any conflicts.
You should look into Lambda idempotency—it’s all about ensuring that repeated operations produce the same outcome. Keeping track of the records you’ve already processed is key, and tools like AWS Lambda Powertools could be quite helpful for managing this.
Thanks for the tip! Adding 'idempotency' to my cloud vocabulary will definitely help with searches.
But what happens if the progress recording itself fails? You've completed the operation but can’t update your tracking record. What would you do then?