I'm building an open-source Kubernetes optimization tool that analyzes Prometheus and OpenCost data to identify opportunities such as CPU and memory rightsizing, idle workloads, and other resource-efficiency improvements. I'm now designing the workflow for turning those recommendations into actual changes.
One option is to update workloads directly through the Kubernetes API, but that would require write permissions in production, which feels risky for an optimization tool. The alternative is a GitOps workflow: recommendation → Git commit → pull request → review and CI → deployment through the organization's existing GitOps controller.
The Git-based approach provides isolation, approvals, audit history, and straightforward rollback, but it also requires integrations for different Git providers, repository layouts, Helm or Kustomize configurations, and environments that may not use GitOps at all.
For production Kubernetes environments, would you prefer direct API changes, automatically generated pull requests, an advisory-only workflow, or a combination depending on the cluster?
4 Answers
I’d make GitOps the default for production changes. Opening a pull request gives teams a clean approval gate, keeps the optimizer isolated from the cluster, and provides an audit trail and rollback path. This can still be useful when the optimizer itself runs inside the cluster; it simply needs to communicate through the repository rather than mutate workloads directly.
The recommendation should include enough evidence for service owners to judge the risk: the observation window, current and proposed requests or limits, confidence, expected cost impact, safety bounds, expiration date, and rollback criteria. Attaching usage graphs or other telemetry to the pull request can make the change much easier to review. Teams are understandably reluctant to reduce resources if they cannot see how stability will be protected.
I especially like the idea of including the historical usage, proposed diff, expected savings, and rollback conditions directly with the change. That should make it easier to evaluate whether the recommendation is actually safe.
A bot with narrowly scoped repository permissions could create the commit, open the pull request, and request the appropriate reviewers. For clusters that are not managed through version control, direct application could remain an explicitly enabled fallback, but I’d avoid making it the default for production. An advisory mode that only reports findings is also useful for teams that are not ready to automate changes.
Automatically requesting reviewers is a good idea because it makes the generated change fit into the team’s normal engineering workflow instead of creating a separate approval process.
I’d support a tiered model: advisory reports first, pull requests for production, and direct API application only when a team explicitly enables it for a sandbox or tightly controlled cluster. Also, don’t consider a recommendation complete when the change is merged. Continue measuring resource usage, cost, performance, and rollback events so the system can confirm whether the optimization delivered the expected result. Policy tools and autoscaling systems can complement this workflow, but they do not replace the approval and audit process for broader changes such as removing unused storage or load balancers.
That matches my thinking. The human approval step and separation from production write access are the main reasons I’m leaning toward pull requests.