I'm working with numerous microservices that are deployed across different environments, and I'm often frustrated by the lack of a centralized view to answer some key questions:
- What version of service X is currently deployed in staging versus production?
- What changes have occurred between these versions—not just commit hashes, but actual business-level changes such as new features or bug fixes?
- When an API gets updated, how do I know which downstream services are affected and might break?
Right now, I have to piece everything together manually from various tools like our CD pipeline, Jira, Git logs, and Slack messages. This method works to some extent, but it becomes chaotic as the number of services scales up. Also, I often don't realize there's a dependency issue with the APIs until something goes wrong.
I've been considering automating some of this by pulling release notes from linked tasks, comparing OpenAPI specs across versions to identify breaking changes, and using Software Bill of Materials (SBOMs) to track which services rely on which internal APIs at which version.
Before diving deep into these ideas, I'm curious about how other teams handle this:
- Do you have a comprehensive system that captures everything?
- Have you explored any specific tools for this (like Backstage, Cortex, Sleuth, etc.)?
- Has anyone successfully built an internal solution that works well over time?
- Or is it just an accepted state of chaos that everyone deals with?
I would love to know what strategies have been effective and what mistakes to avoid, especially for teams managing 20 or more services.
1 Answer
What you're facing is a common issue once you scale up beyond a few microservices. There isn't a perfect solution, but many teams find it effective to use a combination of simpler tools instead of trying to develop a one-for-all system.
First, a basic service catalog is essential where each service is listed alongside its owner, repo, and dependencies—this eliminates the guesswork. Additionally, you should have your CI/CD pipeline report deployment information centrally, so you can see the current versions of deployed services easily.
For tracking changes, auto-generating release notes from commits or linked Jira tickets often works better than attempting to track every detail manually. When it comes to dependencies, things can get messy, and teams may opt for either manual documentation or utilize tracing tools to monitor inter-service calls.
In essence, a straightforward catalog combined with version reporting and basic release notes usually does the trick without over-complicating things. More elaborate systems tend to get ignored or abandoned since they're labor-intensive to maintain.
I like your idea of scanning the repos and using values.xml files to determine current production versions. If you enforce your developers to use Jira tasks, the AI could help generate comprehensive release notes, integrating all that information seamlessly. It seems like a solid approach, and it’s comforting to know we’re not alone in trying to solve this!