I'm looking for advice on how to better implement processing and detection for specific blobs in my Azure Storage account. I have a setup where I created a function app that triggers whenever a new blob is created. It captures details like the container name, the blob's path, and its name. This information is then checked against a hardcoded list of files I'm interested in. I match first on the container name using wildcards, then apply regex to the blob path and name to extract useful data, as these often include timestamps and identifiers.
Currently, I see about 5 to 10 triggers every 10 minutes, but sometimes it can spike to as many as 1000 triggers in a single burst. The implementation has been running smoothly, and subscribers can pick up events to handle them according to their requirements, whether that's for ingestion, re-uploading, or sharing.
I'm curious if anyone has recommendations for alternate implementations or improvements to this setup.
3 Answers
Have you considered using Alerts Rules with triggers? This could help detect new blobs, but you might need multiple rules if you have a lot of files to monitor, which can get costly.
You could implement an Event Grid Topic that subscribes directly to blob events. This way, you could fan out the events to other endpoints, like a service bus queue, which would help buffer the messages and then drain them with Azure Functions as needed.
We are using a regular storage queue right now, but the concept is similar. It’s a common and scalable method that works well for our setup.
Seems like you're hitting a classic XY problem here. Do you have any control over how the files are delivered? I would probably rely on the blob change feed for a more streamlined approach.
That’s an interesting angle. I did try using Alert Rules, but I found it limited me to one rule per file, meaning I'd end up with over 1000 rules. It might still be worth exploring to cut down on function app code!