We're running several Java Spring Boot microservices on EKS, and every service currently uses memory as its Horizontal Pod Autoscaler metric. After looking into JVM behavior, I'm concerned that memory is a poor indicator of actual workload: the JVM can retain heap after garbage collection and may not return it to the operating system quickly. That means memory can remain high when traffic is low, while a heavily loaded service may still have apparently acceptable memory usage.
I'm considering choosing the scaling metric based on the service's workload:
- HTTP request-serving services: CPU initially, or possibly requests per second and p95 latency if demand-based metrics are worth the added complexity.
- Asynchronous queue consumers: queue backlog or age, such as SQS depth, using KEDA.
I'd like feedback on several points:
1. Is this reasoning sound, and is selecting metrics by service type a good approach?
2. For HTTP services, should I start with CPU and move to RPS or latency only if CPU proves to be a poor proxy, or is it worth setting up custom Prometheus metrics immediately?
3. For async workers, is KEDA scaling from SQS backlog the usual approach, or are there better alternatives?
4. How can I verify each service's real workload profile instead of relying only on architecture documentation? For example, can I confirm whether it actually receives HTTP traffic, whether it is truly driven by SQS, and how CPU, memory, garbage collection, throughput, and latency correlate during load tests?
4 Answers
You can verify the workload profile from several angles instead of trusting service descriptions. Check deployment and service configuration for ingress, ports, health endpoints, queue names, consumer environment variables, and sidecars. Compare ingress or load-balancer request metrics with application request counters, and compare SQS backlog and receive/delete activity with pod processing rates. During a controlled load test, graph CPU, allocation rate, heap usage, GC pause time, throughput, error rate, and tail latency together. That will show whether CPU is a reliable proxy or whether a service is limited by garbage collection, I/O, downstream dependencies, or another resource.
The overall direction makes sense. JVM memory is often a poor HPA signal because the heap can stay allocated after GC, so it may behave like a high-water mark rather than a measure of current demand. CPU is a reasonable low-complexity starting point for HTTP services. Add RPS or another custom metric if testing shows CPU does not track throughput or saturation well. For queue consumers, KEDA with SQS backlog is a common pattern.
Before changing the HPA, make sure the JVM heap settings are compatible with the container memory limit. Otherwise, you may replace scaling problems with OOMKills. It is also worth running the new metric in parallel with the current one for a while and comparing when each would have scaled.
Also inspect JVM runtime metrics during both sustained and spiky load. Pod-level CPU can look normal while garbage-collection activity causes poor tail latency. For distinguishing HTTP and async services, check actual connection behavior too: inbound listeners and ingress traffic are strong HTTP indicators, while outbound SQS connections and queue-consumer metrics provide evidence that a worker is really queue-driven.
For HTTP workloads, CPU is usually the easiest baseline because it requires little additional infrastructure. RPS can be a better demand signal when request cost is fairly consistent, but it needs careful configuration: a single target RPS may not work for endpoints with very different workloads. Latency is useful as an alert or guardrail, but scaling directly on latency can be noisy because latency often indicates that the service is already under pressure. Test CPU against throughput, error rate, and p95 or p99 latency before adding custom metrics.
Queue depth, and sometimes queue age, is the standard signal for asynchronous consumers. KEDA can scale SQS-backed workers based on backlog, with limits on minimum and maximum replicas. Make sure the target backlog is based on how quickly one worker can process messages and how much delay users can tolerate. If processing time varies widely, queue age or a throughput-based target may be more useful than raw message count alone.
It can also help to use separate deployment templates or Helm chart defaults for HTTP services and queue workers. The HTTP template can include ingress-related settings and a CPU or request-based HPA, while the worker template can include KEDA and queue configuration. Keep the defaults overridable, since some services will still need a custom metric.