We're running several Java Spring Boot microservices on EKS, and every service currently uses memory as its Horizontal Pod Autoscaler metric. After looking into JVM behavior, I'm concerned that memory is a poor indicator of actual workload: the JVM can retain heap after garbage collection and may not return it to the operating system quickly. That means memory can remain high when traffic is low, while a heavily loaded service may still have apparently acceptable memory usage.
I'm considering choosing the scaling metric based on the service's workload:
- HTTP request-serving services: CPU initially, or possibly requests per second and p95 latency if demand-based metrics are worth the added complexity.
- Asynchronous queue consumers: queue backlog or age, such as SQS depth, using KEDA.
I'd like feedback on several points:
1. Is this reasoning sound, and is selecting metrics by service type a good approach?
2. For HTTP services, should I start with CPU and move to RPS or latency only if CPU proves to be a poor proxy, or is it worth setting up custom Prometheus metrics immediately?
3. For async workers, is KEDA scaling from SQS backlog the usual approach, or are there better alternatives?
4. How can I verify each service's real workload profile instead of relying only on architecture documentation? For example, can I confirm whether it actually receives HTTP traffic, whether it is truly driven by SQS, and how CPU, memory, garbage collection, throughput, and latency correlate during load tests?
3 Answers
The overall direction makes sense. JVM memory is often a poor HPA signal because the heap can stay allocated after GC, so it may behave like a high-water mark rather than a measure of current demand. CPU is a reasonable low-complexity starting point for HTTP services. Add RPS or another custom metric if testing shows CPU does not track throughput or saturation well. For queue consumers, KEDA with SQS backlog is a common pattern.
Before changing the HPA, make sure the JVM heap settings are compatible with the container memory limit. Otherwise, you may replace scaling problems with OOMKills. It is also worth running the new metric in parallel with the current one for a while and comparing when each would have scaled.
For HTTP workloads, CPU is usually the easiest baseline because it requires little additional infrastructure. RPS can be a better demand signal when request cost is fairly consistent, but it needs careful configuration: a single target RPS may not work for endpoints with very different workloads. Latency is useful as an alert or guardrail, but scaling directly on latency can be noisy because latency often indicates that the service is already under pressure. Test CPU against throughput, error rate, and p95 or p99 latency before adding custom metrics.
Queue depth, and sometimes queue age, is the standard signal for asynchronous consumers. KEDA can scale SQS-backed workers based on backlog, with limits on minimum and maximum replicas. Make sure the target backlog is based on how quickly one worker can process messages and how much delay users can tolerate. If processing time varies widely, queue age or a throughput-based target may be more useful than raw message count alone.
Also inspect JVM runtime metrics during both sustained and spiky load. Pod-level CPU can look normal while garbage-collection activity causes poor tail latency. For distinguishing HTTP and async services, check actual connection behavior too: inbound listeners and ingress traffic are strong HTTP indicators, while outbound SQS connections and queue-consumer metrics provide evidence that a worker is really queue-driven.