How would you start rightsizing a Kubernetes cluster without risking outages?

November 4, 2026 3 answers
Asked by sprocket

We have about 400 pods across two environments and already monitor CPU and memory. Some workers request around 20 GB of memory but usually use less than 10 GB. I’m wary of cutting resources when restarts can affect the business, and we’ve also seen exit code 137 without an OOMKilled status. How would you approach rightsizing and autoscaling?

Answers

m1ra-9

I’d treat this as a gradual rollout, not a cluster-wide cleanup. Pick a few low-risk services, agree with their owners on what “healthy” means, and look at a representative period that includes busy times and normal traffic. Make small changes, watch latency, errors, restarts, and resource use, then expand once the results are clear.

For the 20 GB workers, a peak below 10 GB is useful evidence, but it isn’t by itself a safe new limit. Leave room for workload spikes and investigate the 137 exits before making a big cut. Showing teams that a measured change stayed healthy is usually more convincing than asking them to trust a recommendation.

Reply to this answer
OHHEY!

Separate the request from the limit. A request helps the scheduler place the pod and is part of how resources are accounted for. A memory limit is a hard ceiling, so setting it too low can turn an unusual workload spike into an OOM kill. I’d be especially cautious with memory limits and make changes in steps.

Also, 137 means the process received SIGKILL. OOM is one possible cause, not the only one. Check the pod’s termination reason and events, plus relevant node or container-runtime logs, before treating every 137 as proof that the memory limit is too low. If restarts carry serious business risk, understanding that failure mode should come before optimizing the allocation.

Reply to this answer
pocketlint

For a first pass, a recommender such as KRR or Goldilocks can help flag workloads whose configured resources look far from observed use. I’d use the output as a shortlist for review, not as an instruction to apply new values automatically. Monitoring windows, bursts, and the cost of a restart still need human judgment.

For scaling, use HPA when the scaling signal and behavior are straightforward. KEDA can fit event-driven metrics or cases where scaling to zero is needed. Autoscaling won’t make an unsafe memory limit safe, though, and it’s worth checking that adding replicas actually helps the application before rolling it out.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions