Azure ML Job Ran for 70 Days and Triggered Unexpected Compute and Defender Costs
Asked by
MellowQuokka47
An Azure Machine Learning job in an enterprise subscription normally ran for less than 30 minutes, with unusually long runs reaching only 8–12 hours. One job appears to have become stuck and continued running for roughly 70 days, despite Azure ML documentation listing a 21-day maximum job lifetime.
The associated VM generated continuously updated log files in the Storage Account. About 53 days into the run, Defender for Storage was enabled across the subscription with its default 10 TB monthly limit. Defender repeatedly scanned the growing log files, resulting in approximately €1,000–€1,500 in Defender charges, while the VM added another €200–€500. The job and related activity stopped after the Storage Account diagnostics and Defender cap were changed, although the precise cause remains unknown.
Most job logs later disappeared. The remaining ComputeRecord file records the VM size and enough core-seconds to support the roughly 70-day duration. Billing records and resource activity data also remain available.
A Severity B Enterprise support case was opened within the relevant retention period, but the case was initially handled by the Storage team. The Azure ML specialists were involved more than a month later, after the platform's approximately 30-day backend diagnostic logs had expired. Support now says it cannot investigate or consider a refund without those logs, despite the billing evidence, the remaining ComputeRecord file, and the documented 21-day lifetime limit.
The Defender configuration was arguably an avoidable customer-side issue, since no paths were excluded and no cost alerts caught the spike. However, the unusually long VM usage appears connected to an Azure ML failure to enforce its stated limit. Is it reasonable to pursue a refund for the compute incurred after day 21, and possibly the Defender charges that resulted from the prolonged activity? Also, what is the most effective way to escalate this through an Enterprise account manager, TAM, or similar channel when standard support says the missing backend logs prevent further action?
Answers
I’d separate the two issues. The strongest claim is for the VM usage beyond the documented 21-day maximum, assuming the documentation can reasonably be treated as an enforced limit rather than just a quota. The Defender scans appear to have operated as configured, and the missing exclusions or alerts make a full refund of those charges harder to justify. I’d involve the Enterprise account manager or TAM and present the timeline, billing records, ComputeRecord evidence, activity logs, and the relevant limit documentation in one concise escalation package.
Reply to this answer
There are also clear governance gaps on the customer side: no budget alerts, no daily anomaly detection, and Defender enabled without excluding high-churn log paths. Those points may weaken a request for all charges to be refunded, because Defender appears to have scanned data according to its configuration. Still, they don’t necessarily excuse a service from exceeding a documented maximum lifetime. A reasonable position would be to accept responsibility for the normal 21-day period and the Defender configuration, while requesting relief for compute that accumulated after the stated limit.
SilverMarten53: The missing alerts are a fair criticism, but the normal jobs were minutes or hours long, so a 70-day run was far outside the established pattern. That is why the platform limit and the unusually prolonged compute usage seem more relevant than ordinary cost-monitoring failures.
Reply to this answer
Front-line support often follows a strict “no backend logs, no root-cause confirmation” process, so continuing to argue the same ticket may not get far. An account manager, TAM, or customer success contact may have access to a billing-dispute escalation path that ordinary support does not. Keep the request focused and commercial: identify the documented limit, show the exact excess compute period, explain when the case was opened, and point out that the evidence expired during Microsoft’s internal routing delay. Ask for a review of the post-limit compute charges first, rather than leading with every downstream Defender cost.
Reply to this answer
I’ve seen a similar cloud-control-plane problem where a compute resource remained active after its workload state was lost. Support initially requested logs that had already aged out, then rejected the claim. Escalating through the enterprise TAM with a single evidence document eventually produced a partial credit for compute beyond the platform’s stated limit. That doesn’t guarantee the same result here, but the approach is worth trying. I would avoid making the suspected relationship between the Defender configuration change and the job stopping a central argument unless you can prove it.
Reply to this answer
Add your answer
Answers are reviewed before publication.
MellowQuokka47: That distinction makes sense. I’m prepared to concede that Defender was configured on our side, but I still think the compute after day 21 is a much clearer platform-related claim. The main frustration is that support’s delay caused the diagnostic evidence to expire.