I'm building an open-source Kubernetes optimization tool that analyzes Prometheus and OpenCost data to identify opportunities such as CPU and memory rightsizing, idle workloads, and other resource-efficiency improvements. I'm now designing the workflow for turning those recommendations into actual changes.
One option is to update workloads directly through the Kubernetes API, but that would require write permissions in production, which feels risky for an optimization tool. The alternative is a GitOps workflow: recommendation → Git commit → pull request → review and CI → deployment through the organization's existing GitOps controller.
The Git-based approach provides isolation, approvals, audit history, and straightforward rollback, but it also requires integrations for different Git providers, repository layouts, Helm or Kustomize configurations, and environments that may not use GitOps at all.
For production Kubernetes environments, would you prefer direct API changes, automatically generated pull requests, an advisory-only workflow, or a combination depending on the cluster?
2 Answers
A bot with narrowly scoped repository permissions could create the commit, open the pull request, and request the appropriate reviewers. For clusters that are not managed through version control, direct application could remain an explicitly enabled fallback, but I’d avoid making it the default for production. An advisory mode that only reports findings is also useful for teams that are not ready to automate changes.
I’d support a tiered model: advisory reports first, pull requests for production, and direct API application only when a team explicitly enables it for a sandbox or tightly controlled cluster. Also, don’t consider a recommendation complete when the change is merged. Continue measuring resource usage, cost, performance, and rollback events so the system can confirm whether the optimization delivered the expected result. Policy tools and autoscaling systems can complement this workflow, but they do not replace the approval and audit process for broader changes such as removing unused storage or load balancers.
Automatically requesting reviewers is a good idea because it makes the generated change fit into the team’s normal engineering workflow instead of creating a separate approval process.