Hey everyone! I'm relatively new to our DevOps team, having less than a year of experience after transitioning from a development role on the same project. I'm eager to expand my knowledge and improve our deployment and release process, which involves over 50 services. Currently, we're using GitLab CI, and for every release, we manually trigger the deployment pipelines for each service across multiple RHEL servers. With the influx of new services, I feel this approach may not be sustainable long-term. I'm curious about best practices or solutions that seasoned DevOps professionals employ to manage such a complex setup. Any tips would be greatly appreciated!
5 Answers
To kick things off, I'd recommend engaging with your team members to better understand what each service does and the technologies involved. Are they all the same type of service or do they differ significantly? Gathering this info is crucial. Then, you can look at ways to optimize the deployment process. If they're fairly similar, a centralized pipeline might be great. If they vary widely, consider creating tools, templates, and providing guidelines to help your team deploy more smoothly. Just remember, incremental improvements are usually the way to go instead of overhauling systems all at once.
Have you considered using a config server? It can centralize configuration management and make deploying multi-service environments smoother.
For a setup with many services, you might want to look into solutions like Ansible, Nomad, or Octopus Deploy if you're on bare metal. If you’re using Kubernetes, tools like ArgoCD or Spinnaker could simplify your process and enhance orchestration.
When you mentioned manual deployment pipelines for each service, does that mean you have to deploy everything whenever one needs to be updated? It could be worth checking if this is an issue with your CI setup or infrastructure, or even a byproduct of a monorepo setup. Ideally, decoupling your services means you shouldn’t need to deploy them all at once. Automating deployment triggers can help, but ensure it's done thoughtfully instead of just scaling up a problematic solution.
A major win with that many services would be to standardize your pipeline with GitLab CI templates. Instead of having 50+ unique pipeline configurations, create a shared template that each service pulls from. Keep it simple with just the service name and minimal configurations. Environment-based triggers can automate deployment to staging on merges and require just one manual confirmation for production releases. This way, rather than clicking through each service, you'll be managing everything much more efficiently as your service count grows.
We’re actually in the process of implementing GitLab CI templates with Ansible for our configs. The manual deployment is really time-consuming for higher environments, so I’m thinking about your environment trigger suggestion.
Thanks for your response! I totally get that we can't go into all details here. However, our architect is hesitant about bringing in new tools to streamline our manual deployments, which is definitely a pain point for me.