I'm exploring ways to reduce resource usage and scale workloads efficiently in production Kubernetes clusters. However, I've noticed that some cost-saving suggestions may unintentionally conflict with security policies, such as pod security standards, RBAC rules, or resource limits. I'm interested in how others manage this balance. Do you manually review optimization suggestions before using them? Are there automated methods to ensure security compliance along with cost savings? What tools or strategies have you found effective for minimizing risks while optimizing spending? It would be great to hear any real-world experiences or approaches, especially if you've had to make trade-offs between costs and security on a large scale.
5 Answers
Optimizations often get tricky when they suggest things like switching to Spot Instances. For instance, one recommendation to switch a payment API deployment to Spot instances projected savings of $720/month. However, our payment workloads require guaranteed uptime due to compliance regulations, and Spot instances can be interrupted. This could lead to failed transactions and potential audit issues, highlighting how cost-saving moves can create security risks if not carefully evaluated against existing policies.
Here's an example of a cost optimization that backfired for us: we tried to optimize a postgres sidecar and it suggested reducing the memory request from 2Gi to 512Mi. We applied this in dev and staging, and it worked well, but when we moved it to production, our PodSecurityPolicy blocked it due to requiring guaranteed QoS. This misalignment cost us time and trouble during deployment since we had to pause everything to resolve the issue. Another scenario involved multiple optimizations across a namespace; individually, the changes seemed fine, but collectively they exceeded the namespace ResourceQuota, leading to deployment failures. I think we definitely need a better way to assess recommendations against policies.
Totally relate! Manually checking each suggestion can be overwhelming, especially with multiple deployments. Has anyone found a good solution?
A common issue with cost-savings recommendations is that they often suggest changes to resource limits or requests that can conflict with existing RBAC rules. For instance, we used Goldilocks for VPA recommendations, which proposed reducing memory requests from 2Gi to 512Mi for some sidecars. However, because our namespaces have ResourceQuotas and the app teams lack permission to modify those quotas, they faced deployment blocks when applying those optimized settings. We had to get the platform team involved to make the necessary quota adjustments. Also, many recommendations, like bursting QoS, might work in development but fail in production due to enforced policies that require guaranteed QoS for certain services. It’s important to check existing policies before following those suggestions.
That totally makes sense! I’ve faced similar problems with pod security policies when trying to implement changes suggested by cost optimization tools. It’s frustrating when they work in lower environments but hit blockers in production.
We faced a similar deadlock with VPA recommendations. Initially, we tried manual reviews for every suggestion, but that quickly became unsustainable. We shifted our approach by performing optimizations in the PR stage instead of resizing live pods, avoiding runtime security conflicts. As a result, we built a CLI tool to automate this process which helped us catch potential issues before hitting production. The tool is open source if anyone's interested in how we tackled that logic! [GitHub Link](https://github.com/WozzHQ/wozz)
Definitely a frustrating area! Sometimes I feel like these tools don't fully account for our settings, and it takes too long to catch errors once they've already forced a rollback. Have you looked into any new tools that might help?
That’s a great example! It really shows the need for a thoughtful approach to cost optimizations that takes security into account.