How to Set Resource Requests and Limits in a Multi-Tenant Kubernetes Cluster
Asked by
TechieTurtle92
We're managing a multi-tenant Kubernetes cluster with about 15 namespaces for different teams, each deploying their own workloads, which causes a lot of variability in resource consumption. I'm looking for effective strategies to set resource requests and limits without being overly restrictive.
We're currently using LimitRange objects for each namespace, but the teams often find the defaults aren't fitting their needs.
When it comes to CPU limits, should we just stick with requests for scheduling, or is it necessary to enforce limits in a shared cluster? I've come across mixed opinions about CPU throttling potentially leading to more issues than benefits.
Additionally, should we implement ResourceQuota at the namespace level with strict limits, or would it be better to follow VPA suggestions for each deployment? Do people have experience using both?
For memory, we generally set limits equal to requests to make OOMKills easier to troubleshoot, but I'm curious if this is a solid approach or if there are other patterns worth considering.
Lastly, any recommendations for tools to audit and enforce these policies consistently across namespaces? We've looked at Kyverno, which seems promising, but we haven't fully adopted it yet. I'd love to hear how others have tackled these challenges at scale.
Answers
I’d recommend using Goldilocks alongside Kyverno. Goldilocks can provide you with VPA-backed recommendations tailored for each namespace, so you aren’t just guessing on request values. Kyverno can help enforce policies ensuring no workloads are deployed without setting requests and limits. It’s also good to have LimitRange objects in place as a fallback. Just remember that Goldilocks defaults to recommendation mode, so you’ll need a way to actually apply those values, whether manually or through a CI/CD pipeline.
Reply to this answer
Absolutely! Goldilocks works great for recommendations, but keep in mind that if your teams are creating and deleting namespaces frequently, you'll want to watch out for VPA objects piling up. One approach would be to set up a cleanup job or a Kyverno policy that deletes Goldilocks VPA objects when a namespace is removed. You could also limit Goldilocks to only watch certain namespaces by applying specific labels, which helps manage the clutter, especially in environments that see a lot of churn.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions