Hey there! 👋 I've been diving into how teams maintain release governance within Kubernetes environments prior to launching CI/CD efforts. Many teams seem to primarily depend on whether their tests pass, but that doesn't always ensure the actual cluster health before a release. For instance, a deployment might go through successfully, even if the cluster is showing uneven performance with unstable pods or nodes having issues. I've tried out a prototype pipeline designed to check for release readiness across various levels. It includes elements like automated tests with Allure reports, security scans (using tools like Semgrep, Trivy, and Gitleaks), SBOM generation along with vulnerability scanning, Kubernetes readiness checks, and a decision engine that determines if a release gets the green light, should be put on hold, or is a no-go. The pipeline assesses a range of signals, including node status, pod crashes, risk of restarts, and overall cluster health. Everything is compiled onto a single release governance dashboard that brings together results from testing, security scans, SBOM checks, and cluster health validations. I'm eager to learn how others manage release governance in their Kubernetes setups. Do you strictly stick to CI/CD checks, or do you also make sure the cluster is validated before any deployments?
4 Answers
I think your idea is pretty innovative! Many pipelines just push code after passing tests. It’s definitely smart to consider the health of the Kubernetes cluster beforehand. Incorporating this concept into automated workflows could enhance release governance even further.
Have you seen anything like this in action with production pipelines? I'm curious about implementing similar gating mechanisms.
In my experience with various environments, we also check platform metrics, like cluster health and any crashloops, before CI/CD processes begin. Just because the pipeline's green doesn't always mean the environment is safe for deployment. It's crucial to interpret those infra states meaningfully to influence release decisions.
I totally get what you mean! It was actually the reasoning behind my prototype. Many teams rely solely on a green CI status, but the underlying platform might still be facing issues. Discovering a way to incorporate those signals into a GO, HOLD, NO-GO decision could change the release game.
Yes! I think combining those checks and metrics can help legitimize the release decision process. How do teams you’ve worked with generally define what those platform checks look like?
Interesting point! It seems like your system could actually help in incidents rather than hinder them. The idea of giving GO, HOLD, or NO-GO is neat because it gives teams the chance to assess risks, but how do you handle manual overrides during emergencies?
Great question! In the prototype, if we hit HOLD or NO-GO, it's not an outright block. Teams can still apply manual overrides in critical situations like hotfixes. The goal is to flag risks rather than create a brick wall for deployments.
Exactly! We need to make sure that during incidents, the team still has the flexibility to push urgent fixes. That’s why signals about the state of the infrastructure should be part of the release conversation.
Relying on just 'checking the vibes' of the cluster can be risky. What kinds of signals are you using that actually matter? If one node isn't doing well, does that mean a release should automatically get blocked? I'd prefer having clear policies in place, like leveraging OPA or Kyverno, along with progressive delivery instead of just relying on how the cluster feels.
Preflight astrology is a hilarious way to put it! Honestly, this seems more like a solution searching for a problem. If your cluster is in bad shape, then the platform team needs to get that sorted out before any deployments should happen.
You make a solid point! The intent isn’t to halt releases over minor node issues, but to ensure that significant stress is identified before adding to the chaos. Ideally, we would be setting clearer policies and thresholds based on real metrics.
Thanks! That was the entire purpose of creating the prototype. A lot of systems currently operate just on passing tests, but they neglect the real risks of cluster health. Integrating this into workflows would definitely make it a standard part of the release process.