Hey everyone! I'm looking for an efficient way to calculate the total count of blobs and their sizes for each container in my Azure storage account. Some of my containers have over a billion blobs, so using a simple list-blobs script isn't really a viable option. I'm seeking advice on a method that allows me to retrieve this data either daily or weekly without putting too much load on the storage account. Specifically, I'm hoping to find a solution that can provide: total blob count per container, total size measured in GB/TB per container, and an approach that scales well with massive blob counts—all while being automated for regular use. If you've got insights on using AzCopy, Storage Insights, metrics, or any clever scripts/workarounds, I'd really appreciate your input. Thanks a lot!
1 Answer
You might want to try the Blob Inventory feature. It does a nice job of summarizing the data so that you can run calculations directly. I used it to generate parquet files, and it provided all the info I needed.
But keep in mind, Blob Inventory isn't ideal for billions of blobs. It can be pretty slow and costly to process, not to mention you're dealing with outdated data by the time the inventory arrives.