I manage a relatively small Kubernetes environment with roughly 400 pods across two environments. We're starting an effort to rightsize applications and introduce HPA, KEDA, and affinity rules. We already collect CPU, memory, and other metrics, but I'm hesitant to reduce resource settings because an OOM-related restart could have a serious business impact. Some worker services request around 20 GB of memory even though their observed usage is usually below 10 GB, yet they occasionally terminate with SIGKILL and exit code 137. I've been explaining that exit code 137 only indicates SIGKILL and does not automatically prove the container was OOMKilled, but there's still concern from the development teams. What process would you recommend for rightsizing workloads, choosing between HPA and KEDA, investigating exit code 137, and building confidence before making changes?
4 Answers
For autoscaling, ordinary HPA is usually enough for CPU, memory, or external metrics. KEDA is especially useful when scaling should follow queue depth, consumer lag, pending jobs, schedules, or when scaling to zero is important. For node capacity, use a node autoscaler appropriate to your platform, such as Karpenter or the equivalent in your environment. Keep autoscaler changes separate from rightsizing at first so it’s clear which change affected reliability or cost.
Start with a small number of low-risk services instead of changing the whole cluster. Use a meaningful observation window and review p95 or p99 CPU and memory usage, restart history, traffic spikes, and application behavior. Reduce requests gradually and keep deliberate headroom; be more conservative with memory than CPU. Recommendation tools such as VPA in recommendation-only mode, Goldilocks, OpenCost, or similar systems can provide a useful starting point, but I wouldn’t auto-apply their values at first. Publish the current settings, measured usage, proposed values, and safety margin so each application team can review the evidence.
Keep requests and limits conceptually separate. Requests primarily affect scheduling, HPA calculations, and how workloads compete during node pressure; setting them too high can leave capacity stranded and force unnecessary node scaling. Limits can protect the node, but an overly low memory limit can directly cause an OOM kill, while a CPU limit can introduce throttling. Change one dimension at a time and watch both application performance and node contention rather than setting values solely from a usage graph.
Treat exit code 137 as a symptom, not proof of an out-of-memory kill. It means the process received SIGKILL, which can happen during an OOM event but can also result from other causes. Check the container’s termination reason, pod status, node and pod events, kernel or runtime logs, eviction information, and whether something outside Kubernetes killed the process. Once you can show the actual cause for each restart, the resource discussion becomes much less speculative.
Also address the business impact of restarts before optimizing aggressively. If one restart can cause a major outage, improve graceful shutdown, retry behavior, redundancy, queue durability, and alerting at the same time.
The trust-building part matters as much as the tooling. A few safe wins that reduce waste without causing incidents will make teams much more comfortable with later changes.