I'm dealing with a monthly spend of about 1.1 million, and I've got a year's worth of monthly cost exports stored in a data lake. However, connecting Power BI to this data lake is painfully slow; it just hangs on the Prices and Costs loading screen. I even let it run overnight, but it timed out after around 12 hours due to a network disconnect. Am I doing something wrong? Should I be using a VM in Azure instead of my home laptop? Any tips or documentation that could help me out?
3 Answers
Can you give us a little more detail? What Azure service is your Power BI connected to? Do you have a published report in your Power BI workspace? If so, how long does it take to load? Also, did it ever work from your home network or has it always been an issue?
You might want to look into loading your data into Azure Fabric or Databricks first, which would make it much more queryable. It could really enhance your Power BI experience.
Yep, using Data Factory with a Fabric instance is the way forward. This was just a proof of concept, but it's good to know what the future looks like!
From what I've seen, using blob storage for that amount of data can lead to long load times. Instead, you might want to consider leveraging Azure Data Factory; it significantly speeds things up – we handle around 1.2M a month and it loads in seconds.
I took this advice and upgraded my storage account to the performance tier. I exported a month of costs into it, set the connection to Monthly granularity, and it's working great now! Definitely looking towards adopting Data Factory fully.
As of now, it hasn’t loaded at all from my home network. It appears to be loading, there’s a significant amount of data. I’m using the default CostSummary.storage.pbit from the FinOps toolkit, connecting to a Gen2 Data Lake using an access key.