How can I effectively track interdependencies between repositories in a large organization?
Asked by
TechieExplorer42
I'm part of an infrastructure automation team in a large organization with hundreds of repositories on GitLab. We create shared Docker images, reusable CI templates, and Terraform modules. One major challenge we're facing is managing dependencies, especially when a breaking change occurs in a shared resource and affects multiple repositories. Currently, most of this knowledge is held by a few senior engineers, leaving newcomers confused and unaware of the impacts of their changes.
We've explored GitLab's dependency scanning, but it's more focused on identifying vulnerabilities in external packages rather than managing dependencies between our internal repositories. We've also looked at tools like Backstage, but manually creating YAML files for every dependency seems inefficient given our scale.
I'm curious to hear how other organizations manage this issue. Do you have any internal tools, spreadsheets, or do you just deal with the fallout as it happens?
Answers
One effective strategy is to maintain versioning for all shared components like Docker images and Terraform modules. Instead of using the `latest` tag, each repository can pin to a specific version. This way, a breaking change won't disrupt everything at once. Additionally, some teams automate the creation of a dependency map by scanning their repositories to identify common dependencies. Though not perfect, it provides a decent overview of dependencies and should help mitigate sudden issues when something breaks.
Reply to this answer
In theory, maintained versioning should avoid breaking changes from crashing everything. However, in practice, CI templates and shared Docker images often end up being tracked by mutable branches like `main`, which is what leads to sudden failures when someone pushes a breaking change. No one expects a simple push to break their pipeline! It would help if there were a clearer method to visualize who depends on what, especially when planning a significant update.
Reply to this answer
An interesting approach might be creating a mock API that various projects can test against before they hit the integration environment. By versioning this mock API and ensuring that each team contributes to its upkeep, you can have a safety net to see which project versions have been tested against specific mock versions.
Reply to this answer
If you're open to it, adopting a monorepo strategy can really streamline dependency management. With a monorepo, you can test all affected components in one go whenever a change is made—this can prevent the chaos caused by breaking changes across multiple repositories. We follow a strict rule of determinism, ensuring all external dependencies are pinned to specific versions. It's worked out well for us, making the release process much smoother.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions