When the Horizontal Pod Autoscaler adds or removes application or database pods, existing long-lived connections can remain attached to the original pods while newly created pods receive little traffic. Does HPA expose any native lifecycle event or hook that applications can use to invalidate or gradually rebalance connection pools after a scale change? If not, what is the recommended approach: pod lifecycle hooks, watching endpoint changes, finite connection lifetimes, load-balancer settings, or a database-aware proxy?
4 Answers
A database-aware proxy is often the cleanest option when many application replicas connect to the same database. The pods connect to the proxy, and the proxy manages a smaller, shared set of database connections. Examples include PgBouncer or PgCat for PostgreSQL and managed database proxies for supported cloud databases. This is continuous pooling rather than a one-time action triggered by a new replica.
Be careful with pool sizing during scale-out. A pool configured for 10 connections per pod means 3 replicas use about 30 connections, while 20 replicas may try to open 200. That can exhaust the database before connection staleness becomes an issue. Keep the per-pod pool small and account for the maximum replica count when setting the database connection limit.
HPA only changes the replica count; it has no awareness of application connection pools and does not emit a connection-rebalancing event. The application should manage this itself, usually by setting a finite maximum connection lifetime and idle timeout so connections are rotated naturally. Watching service endpoint changes can also work, but it adds complexity and usually isn't necessary.
Load balancers and service proxies can distribute new requests to new pods, but existing keep-alive connections may still stay where they were established. Check connection keep-alive and draining settings, and use graceful termination so a pod being removed stops accepting new work before it exits. That complements connection lifetime limits, but it isn't an HPA-specific rebalance hook.
A proxy can handle the same problem when changing the application isn't practical, but the pool still needs sensible limits.