Hey everyone! I'm relatively new to our DevOps team, with less than a year of experience, transitioning from a development role on the same project. I'm looking to broaden my knowledge and came across the challenge of managing deployments and releases for a project that consists of over 50 services. Currently, we use self-hosted GitLab and GitLab CI, and the deployment pipelines for each service are initiated manually whenever there's a release for an environment. We have multiple RHEL servers for each environment, but I'm starting to think that this manual process won't scale well as we introduce more services. How do seasoned DevOps professionals typically approach this kind of deployment and release strategy?
5 Answers
First off, it's crucial to really understand what each service does and their tech stacks. Are these services similar, or are they all unique? Understanding their individual requirements and deployment processes is essential. Try connecting with the teams to hear their inputs. With that knowledge, you can then develop a platform that helps these teams deploy faster and safer. If the services are similar, you might consider a centralized pipeline. If they differ significantly, focus on providing tools and patterns that teams can use effectively. Incremental improvements often work better than sweeping changes!
If you're on bare metal, consider using Ansible for orchestration, or tools like Nomad or Octopus Deploy. For Kubernetes environments, look into ArgoCD or Spinnaker to streamline your deployments. These technologies can automate a lot of the manual work you’re currently doing!
Have you thought about implementing a configuration server? It could help manage configurations across your services more efficiently, reducing some of the manual burdens.
A game changer for you might be standardizing with GitLab CI templates. Instead of juggling 50+ separate pipeline configurations, create a shared template for all services. You can set up minimal configurations that each service can inherit. This way, merges to the main branch can auto-deploy to staging, and a single manual approval can trigger production releases, instead of managing each service individually. This will save a ton of time as your project grows!
We're actually beginning to implement GitLab CI templates along with Ansible, but the manual deployment in the higher environments is still a sore spot for me. I think I need to explore those environment triggers further.
When you mention needing to deploy each service manually, do you mean that deploying one service requires deploying all of them? It's worth asking whether this is a limitation of your CI system or if it's more about your current setup. Ideally, you’d want to decouple these services so they can be deployed independently. Automating 50 deployment triggers seems like a solution, but maybe reconsider that if there's a broader underlying issue with how things are structured right now.
Thanks for the tips! I realize that gathering detailed requirements is tough in our current culture. Our Architect is resistant to adding tools to reduce the manual clicks during deployment, which really makes things challenging for us.