I'm facing a unique challenge with managing CI/CD access between my GitHub organization and several AWS accounts. I have over 80 repositories that need to interact with around 8 to 10 different AWS accounts, each serving distinct purposes like security, logging, etc. We have set up a deployment account as the only authorized entry point for the pipelines.
The main constraints are that not all repositories should have access to all accounts; each repository should only access its relevant account. Additionally, the provisioning roles that the pipeline assumes must adhere to least privilege policies, and I'm looking for a scalable solution that doesn't require manual operation for onboarding such a large number of repositories.
I'm considering a setup where GitHub Actions connect to an OIDC role in the deployment account, which then assumes provisioning roles in the respective workload accounts.
The core of my challenge is designing the control plane:
- Where should the mappings of repositories, environments, and accounts be stored?
- Who is responsible for creating and managing these roles?
- How can we scale the onboarding process for 80+ repositories without extensive manual adjustments for IAM?
- How do we ensure the workload roles remain least-privileged without creating an unmanageable setup for each repository?
I envision a central repository that would manage all IAM and trust relationships through a declarative approach, allowing application repositories to consume pre-created roles. With this in mind, I'm less focused on how to assume a role from GitHub and more on how to effectively design this central access-management layer.
6 Answers
I recently implemented a solution using GitHub Actions where Vault validates JWT tokens from the OIDC provider so that it can assume roles in AWS. The key is that Vault checks the repo name from the JWT to ensure that only allowed repos can assume roles. You can automate role provisioning with Terraform based on a centralized mapping configuration that you trigger via pull requests. When the PR is merged, Terraform can be applied to set up the roles accordingly.
Check out esseesseff.com. They provide patterns and solutions that might align well with your control plane requirements and can help in addressing those challenges you've mentioned.
You could dive into OIDC AWS ROLE based access at the pipeline level. Consider creating role policies that differentiate between branches, allowing each repository to assume its designated role with specific permissions tailored for that account. However, the scalability is a concern—how do you handle creating these roles across multiple accounts? Have you looked into the 'jump role' strategy for CI/CD to manage permissions across your deployment accounts?
Exactly! It's about maintaining a balance between usability and security. If you're thinking about automated role creation, you'll need a script or process that runs on initial deploy to set up those roles without requiring manual access to each account.
You might want to consider using Terraform to automate this. It helps with managing the IAM roles and permissions needed across multiple accounts seamlessly. Check out the guides on using GitHub Actions with AWS OIDC for setting up secure gateways for your pipelines. But keep in mind the chicken and egg problem: how does Terraform initially access those accounts without the pipeline permissions? That’s a crucial aspect to address!
That's a tricky situation, for sure! I think figuring out the initial permissions is vital. Maybe you could set up a temporary role that has broader access just for the Terraform setup phase?
Not to be rude, but how sure are you that you're on the right track here? What do you mean by all these repos needing access to AWS? Can you clarify that?
Absolutely! Each of the 80 repositories contains Terraform setups that need to deploy resources but to different AWS accounts for varied purposes. It’s about managing access securely while maintaining efficiency!
A smart CI/CD setup will definitely help you address these problems. Leveraging OIDC alongside EC2 instance roles can streamline your workflows. A well-thought-out architecture can solve many headaches, so make sure your CI/CD solution supports this kind of configuration!
Are you referring to HashiCorp Vault? I’d love to use that, but the costs for licenses might be a hurdle for my corporate environment. Did you find ways to bootstrap repositories efficiently?