I've been experimenting with running Kubernetes policy checks earlier in the process, specifically in pull requests before they get merged. The goal is to provide developers with quick and clear feedback without the delays of waiting for pipelines or needing access to a cluster. Recently, I added support for Open Policy Agent (OPA) using WASM so that these policies can run completely offline during the review flow. I'm interested in how others approach this situation:
- Do you rely solely on CI or admission checks?
- Have you implemented validation during the PR stage?
- What strategies have you found to be effective or ineffective in practice?
5 Answers
I'm trying to address a persistent developer experience issue. My current setup still has a lot of manual checks at the PR level. Policies are constantly changing, and many engineers don’t even realize they’ve altered until it’s too late. When violations appear post-merge, it’s a hassle – you waste time digging through logs and going through multiple rounds of fixes, all while admission control blocks deployment for unrelated reasons. It feels like a drawn-out guessing game instead of receiving prompt feedback.
I recommend running checks at the PR stage and enforcing rules at the admission level. This way, you catch issues early and streamline the developer experience.
While I don't work directly with Kubernetes, we do a ton of checks right during PR creation, including automated comments that need to be acknowledged or resolved before anything gets merged. It's worked wonders for us; we trust the system so much that merges go straight to production automatically without any issues. The key takeaway here is to do policy checks as early as possible for quicker feedback loops and to implement a 'shift-left' strategy in your workflow.
I'm tasked with setting up something similar at my job. Just wondering, what CI tools do you use for your setup? If it's public, I would love to check it out!
I think all three stages have their merits, but without enforcement in admissions, your checks might not be effective. It's about where you get the quickest feedback and strong enforcement:
- Implement checks during the PR/CI phase for immediate insights without cluster access. Tools like OPA Gatekeeper or Kyverno provide robust admission checks that block bad manifests at the API server.
- A hybrid approach that combines early checks with admission controllers helps catch fast issues and maintains a safe runtime environment.
- As for policy engines, Kyverno is easier to adopt since it’s Kubernetes-native. On the other hand, OPA/Gatekeeper allows for more complex, organization-wide policies if needed.
In my experience, blending early feedback with admission enforcement helps balance developer speed with safety.
I totally agree there isn't one perfect solution. I've been trying to fill the gap between the initial coding and checking phases, where developers often lose their context. I developed an open-source tool that runs similar policies directly in the PR, giving instant feedback during reviews. The aim is to speed things up and reduce the back-and-forth, while the actual enforcement continues with admission controllers.
That's a really interesting approach! Solving that gap seems crucial. I’d love to know more about how your tool works in practice!
I've experimented with several setups. In my experience:
- PR checks are fantastic for catching issues early, which keeps the workflow smooth before the review stage.
- CI checks are essential for integration, while admission should enforce the hard rules (like security and disallowed capabilities). However, admission isn’t ideal for ‘style’ rules since those can hinder deployments.
- We handle policy validations early and ensure that admission is mainly focused on critical checks, offering clear guidance on what needs fixing so it doesn't feel too restrictive.
That sounds really impressive! What kind of automation tool are you using for the comments? Do these checks validate against actual policy violations that occur at the cluster level?