I'm currently working as a Software Development Engineer in Test (SDET) and I've recently started learning Terraform. Our team is transitioning our CI/CD pipeline to GitLab while using AWS as our cloud provider, moving away from Azure and Azure DevOps. I'm still getting my head around all of this, so I'm hoping to get some insights from you all.
We plan to use Terraform for managing our infrastructure. I've managed to set up a pipeline that seems to work at a basic level for a .NET Blazor demo app that we're deploying on AWS Elastic Beanstalk. Here's how I've set it up so far:
- GitLab CI/CD is responsible for building and publishing the application to a deploy-version.zip file.
- A deploy job sends the zip file to an S3 bucket using the AWS CLI.
- A Terraform plan job runs on every push, storing the plan as an artifact.
- The apply command is a manual job that makes changes based on the plan, and our tfstate file is stored in S3 with DynamoDB for locking.
I'm wondering if my approach aligns with best practices, specifically:
- Should Terraform handle app deployments and the transfer of the deploy.zip to S3, or is that a separation of concerns issue?
- How often should we really be running Terraform apply?
- Is it advisable to manually set up resources on AWS first and then import them into Terraform, or should I create the resource files upfront?
- Is the primary use of Terraform to ensure resource consistency?
- Many on the team are accustomed to Azure DevOps and think they'll easily transition to GitLab CI/CD pipelines; is that unrealistic?
Also, I've heard about using Helm charts but I believe they're more for Kubernetes (EKS) rather than ECS? I've been told ECS can be quicker and cheaper, while Beanstalk is simpler for apps that don't need rapid scaling. I appreciate any advice you can provide!
3 Answers
Definitely leverage GitLab's native Terraform state handling. It'll save you from a lot of headaches! Also, be cautious with secret management—keep those external to your Terraform setup since they require frequent updates. And don't forget to implement a comprehensive tagging policy for your resources; it’s really helpful for tracking and management.
Your setup sounds decent! Having separate repos for bootstrapping infrastructure, like your S3 for tfstate, is a solid approach. You should be running Terraform plan with every code change and apply on the main branch only to minimize risks. Terraform is designed to handle drift, but you’re right; you don’t want to overuse apply unless necessary.
Exactly! Terraform is great for consistency but keep in mind it shouldn’t be handling all your app deployments—that can lead to messy separation of concerns.
GitLab does have built-in support for Terraform which can simplify things for you, so you might not need to manually manage everything in your CI/CD setup. And yes, Helm is specifically for Kubernetes platforms, not ECS; so don’t get too tangled in that if it's not what you need.
It's true, you can use Helm for any Kubernetes environment, not just EKS. Just saying it's a bit misleading to say it’s only for one platform!
I completely agree! Plus, testing full spins of Terraform can make sure you're prepared for disaster recovery. Just ensure your CI/CD pipeline allows for that without timing out.