I'm curious to hear about the most significant mistakes you've made while working as an AWS Developer or Architect. What happened and what did you learn from it?
5 Answers
While covering for a vacationing coworker, I ended up running about 400 AWS instances for a project, not knowing they weren't supposed to be left idle. After two days, it racked up a $16k bill! I documented everything, but ended up with a negative review. Luckily, I got laid off shortly after and secured a much better role!
I let someone make manual changes directly in production using the console. That was a huge mistake that I regretted later! Definitely not something you should do without proper controls in place.
Management of changes can be tricky, especially in large companies. In smaller setups, is a direct console change as risky? Maybe not!
Once, I was deleting an RDS database in the test environment to run some automation tests. Unfortunately, I accidentally used production credentials and took down the production database instead! Thankfully, I had taken a final snapshot just in case, but it was a nerve-wracking experience. Now I always make sure to double-check my terminal!
These kinds of mishaps can truly happen to anyone, no matter your experience level. Definitely needed some guardrails there!
Wow, what a freak accident! I had something similar happen, but luckily it only cost me a few dollars.
My biggest blunder was spinning up a Managed NAT Gateway. That’s how my crazy adventure in AWS started!
Well, bankrupting your company is usually a fast track to an exit interview!
I had a Lambda function triggered by file uploads that ended up writing back to its trigger path, which created a loop. It ended up costing around $100k because nobody at work fixed it, and it happened five times!
$100k? That's insane! Did AWS give you a break on that, or did your company foot the bill?
How did it escalate to that cost? I've faced similar problems but they didn't hit my budget nearly as hard.
That sounds rough! You did your best given the circumstances, and it's good to hear that it led you to something better.