I'm looking for a way to ingest Azure cost data directly into Databricks without relying on any external tools or services. Is there an API available for this? I'd prefer to use custom code, perhaps utilizing copy activity in Azure Data Factory or the REST API in Databricks to achieve this. Any guidance would be appreciated!
2 Answers
The Azure Cost Management REST API is exactly what you need. Check out the Usage Details endpoint to pull billing data directly into Databricks. You'll have to set up service principal authentication and then call the API in your notebook or ADF pipeline. Just keep in mind that you'll need to manage pagination and handle rate limits since the cost data can get pretty heavy. Also, there’s a service called Pointfive that implements this ingestion pattern really well if you want a cleaner solution.
You could export your cost reports to a storage account and then access that data from there. It's a straightforward way to get your data into Databricks without using third-party tools.
This advice was really helpful, thanks!