I'm building an open-source Kubernetes optimization tool that analyzes Prometheus and OpenCost data to identify opportunities such as CPU and memory rightsizing, idle workloads, orphaned load balancers or persistent volumes, and other resource-efficiency improvements.
The next step is making those recommendations actionable. One option is to update workloads directly through the Kubernetes API, but that would require write permissions and makes me uncomfortable for production environments. The other option is a GitOps workflow: recommendation → Git commit → pull request → review and CI → deployment through a tool such as Argo CD or Flux.
The Git-based approach provides isolation, approvals, audit history, rollback, and compatibility with existing deployment practices. However, it also means handling different Git providers, private or self-hosted repositories, Helm and Kustomize layouts, and organizations that do not use GitOps.
For production Kubernetes, would you prefer direct API changes, automatically generated pull requests, or another workflow? What safety checks and evidence should accompany each recommendation?
4 Answers
I’d strongly favor GitOps for production. Have the tool generate a commit and pull request, then let the normal review, CI, and deployment process handle it. That gives you a clean permission boundary and a built-in human approval gate, even if the optimizer itself runs inside the cluster.
Make pull requests the default, and use a dedicated bot with only the permissions needed to update the relevant files and request reviewers. For environments that are not managed through Git, direct application could be an optional integration, but it should be explicitly enabled per cluster rather than the default behavior. Sandbox clusters are a good place to test direct changes.
I like the idea of making the behavior configurable per cluster: advisory or PR-based in production, with direct application available only for explicitly approved non-production environments.
Keep the system advisory-first. It can produce a report or proposed diff that a team reviews and applies according to its own workflow. Policy tools can also provide useful audit-only signals. For rightsizing specifically, existing autoscaling tools may cover part of the problem, while a broader optimizer can focus on less automated cases such as unused storage, idle services, and orphaned resources.
The quality of the recommendation artifact may matter more than the Git provider integration. Include the observation window, current and proposed values, confidence, expected cost and performance impact, safety limits, an expiry date, and clear rollback conditions. Attach historical usage graphs or similar evidence so service owners can judge whether lowering requests is safe. People are understandably cautious about changes that might affect reliability.
That’s a great point. The pull request should include the usage history, proposed diff, expected savings, and safety rationale rather than just changing a number. We also need to measure the outcome after deployment instead of considering the recommendation complete when the PR merges.
That matches my thinking. The approval and audit trail are the main reasons I’m leaning toward pull requests rather than giving the optimizer production write access.