Hey everyone! I'm diving into a little case study to figure out why people end up with massive cloud bills. Can you share your experiences with the worst cloud bills you've ever received? What triggered those charges? Were they due to a mistake on your part or someone else's? How did you handle the situation afterwards, and have you set up any safeguards to prevent it from happening again?
5 Answers
One time, a co-worker accidentally pushed a root AWS key to a public GitHub repo, and within minutes, we had thousands of GPU instances mining crypto. We shut them down quickly and contacted AWS, who thankfully refunded us and reminded us to be more careful. As for regular expenses, watch out for Datadog; it can get super pricey, especially with logging at scale.
Why was he using the root keys? Why does the root user even HAVE keys?!
As a student tinkering with AWS projects, my biggest killer bill was a NAT gateway I forgot to terminate. It's easy to overspend if you’re not careful!
Same! NAT gateways can quietly drain your wallet without you noticing.
Dude, I ran a massive query in Bigtable that took an hour. The database wasn’t enormous, but the bill sure was! Now I monitor queries closely to avoid surprises.
Do you have any guardrails to monitor compute usage?
I had a team member test Cosmos DB in Azure, but they misconfigured it and ramped up costs to $15,000 overnight! I only caught it because I reviewed the billing the next day. Now we have cost anomaly trackers to prevent future mishaps.
Just reach out to support! I used to work in billing for Azure and saw everything from small refunds to people with hacked accounts mining crypto. My favorite refund requests were things like 'I have no money for milk'. Be honest and fill out their surveys; it’s useful for them!
How often did you get those kinds of complaints? What were the usual cases?
How does stuff like this make it through PR reviews?