I recently contacted GitLab support regarding a security concern I have with deploying AWS commands via GitLab runners using IAM roles. GitLab runners can be thought of as disposable machines, similar to EC2 instances or Kubernetes pods, which assign IAM roles upon job execution. We dynamically spin up these runners as pods with specified permissions using Cloud Development Kit (CDK).
The worry here is that if a developer knows the tag name of the runner in the gitlab-ci.yml file, they could potentially run commands like 'aws sts get-caller-identity' to identify the IAM role and misuse the permissions. This can escalate to running harmful commands such as 'aws ec2 terminate-instances' or 'aws autoscaling update-auto-scaling-group'. Unfortunately, there's a history of having to grant extensive additional permissions just to execute actions like 'cdk deploy', which leads to a wider attack vector.
I'm seeking thoughts on how to mitigate these risks effectively?
4 Answers
I see where the confusion might be. If the GitLab runner is indeed a pod, it's using the pod service account or potentially relying on node permissions. If you granted permissions for CDK, do ensure it includes permissions for STS and EC2. To minimize risk, focus on scoping permissions narrow enough—like just allowing 'sts:AssumeRole' for your CDK deployments. This way, you can ensure its functionality without exposing too much.
One approach you could take is to remove the EKS service account and implement OIDC through your GitLab repository settings. From there, you can enforce stricter IAM role permissions. It's important to acknowledge that any CI/CD setup can inherently be misused by someone with good coding skills, so implementing compensating controls to limit potential damage is crucial. Just be prepared for it; it's a part of how CI/CD operates!
Another way to control access is by adjusting the Instance Metadata Service (IMDS) hop limit to 1. This will ensure that the host can use the instance profile, but containers running in the host will not have access. Coupled with GitLab OIDC, you can allocate IAM privileges on a job-by-job basis, which helps tighten security significantly.
What you're experiencing is pretty standard behavior. To prevent access to the runner's privileges, consider isolating pipelines by using separate repositories or have their repository trigger your pipeline. Alternatively, allowing them to work in a separate dev account can ensure you maintain control, requiring approvals for merges into the master branch.
Thanks for the ideas. I was thinking about triggers today as well.
Thanks for the ideas!