I'm looking for a way to efficiently trigger AWS CodeBuild only once after I've uploaded multiple files to an S3 bucket. I need to implement a setup similar to what was discussed in a previous AWS Re:Post thread, but that thread doesn't provide enough clarity on the implementation. My specific use case involves scanning the files via CodeBuild with ClamAV, and if any are infected, they get moved to a quarantine bucket. The main challenge I'm facing is that I want to start the scanning process just once after all the files are uploaded—not for each file separately.
From what I understand, S3 itself won't trigger CodeBuild, so my plan is to have S3 trigger a Lambda function (potentially through SQS), which would then kick off the CodeBuild process after confirming that all necessary files are uploaded. If anyone has insights, articles, or examples about how to handle batch uploads in this way effectively, I would appreciate your input!
5 Answers
How large are the files you're dealing with? If it's feasible, you could compress them into a zip or tar file. Additionally, with SQS, you can tweak the batch size and window to ensure you capture all files in a single trigger.
Why not restrict uploads to only zip files? You could upload a zip, extract it using Lambda, and process the files from a different bucket after unpacking. Also, have you considered using Metrics and Alarms to trigger your Lambda?
You're right in that you'll need something in the middle to check when all the files are uploaded. Once that happens, you can trigger CodeBuild with the details of the files needing scanning. Also, triggering the Lambda directly without using a queue seems doable.
I've actually implemented a workflow like this before. The first Lambda checks for all expected files by reading a JSON payload that lists them. When all files are there, it can zip them up and trigger another Lambda to handle cleanup and processing. It works well for smaller uploads, typically around 5-40 files a day.
One idea you could try is uploading a 'done.txt' file last, which would act as a trigger for the Lambda function. The Lambda could then read this file to get the names of all the previously uploaded files. It's not a perfect solution, but it might work for your scenario!
Related Questions
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically
[Centos] Delete All Files And Folders That Contain a String