I just joined a new company and was shocked to find that all workloads, from production to testing, are running in the AWS management account without any separation between them. There are no member accounts or guardrails in place. Many old resources are still running, likely left over from testing, and figuring out their necessity for deletion is a headache. To complicate matters, developers have admin access to the management account. I realize this setup is quite risky and I plan to address it with leadership soon.
My main challenge isn't about fixing the organizational structure in one go but rather dealing with the lack of processes to track things like:
- Who owns each resource?
- Why does it exist?
- How long it should stay (especially for non-production resources)?
This leads to wasted costs, confusion during incidents, and risky cleanup decisions. Since the management account doesn't allow for service control policies (SCPs), transitioning everything to member accounts feels daunting.
For others who've faced similar chaotic setups:
- What were your first steps to implement a practical process?
- How did you enforce ownership and resource expiration without using SCPs?
- What minimum standards should DevOps set?
- Did you stabilize the situation first, or did you advocate for account separation right away?
I'm seeking tried-and-true advice, not just theoretical solutions.
6 Answers
Just start by listing all resources. Create a spreadsheet for teams to fill in details like owner, email, product, team, and purpose. Give a strict deadline but keep it reasonable. After the deadline, any untagged resources should be stopped first, and a little later, those still untagged should be terminated.
Sometimes the best way is to start fresh. Set up a new AWS root account and a clear billing structure. You can plan your organization and OUs from scratch while recreating everything in Terraform. It's like cutting your losses instead of rearranging the mess in the current account.
What are the pros and cons of this? I think this approach could work well; should I just ask support for assistance with it?
I started by creating a separate AWS organization with new Organizational Units (OUs) and clearly defining the purpose of each OU. Then, I set up Single Sign-On (SSO) to make access management easier. It helped streamline things from the start!
Thanks for your response, any documentation you have? Like official docs from AWS?
First things first, is anything defined as code? Besides just creating an inventory, putting everything into Terraform with a solid CI workflow for changes is a great start. This really helps enforce structure and accountability. But honestly, it will need a lot of planning and discussions with the team since it's more about changing mindsets than just tech fixes!
Yes, you're right; this is a people problem and nothing is defined as code. I'll definitely talk this through with management. Thanks!
Consider establishing a standard set of tags that all teams need to apply to their resources. This is straightforward and can be easily integrated with Terraform. After that, give everyone a deadline to tag their resources—any untagged resources should be flagged for deletion soon after. This tagging system also helps track spending!
That sounds reasonable. Developers might react differently when I propose this, but I like the idea of tagging. Thanks!
What you really need is FinOps! It's focused on optimizing the financial management of cloud resources. You can start by getting familiar with the concept and how it can benefit your organization.
I'm very new to this term, does it really help? How do I start?
Good idea, thanks!