When the Horizontal Pod Autoscaler adds or removes application or database pods, existing long-lived connections can remain attached to the original pods while newly created pods receive few or no connections. Does HPA expose a lifecycle event or hook that applications can use to invalidate or gradually rebalance their connection pools after scaling? If not, what is the usual Kubernetes approach: lifecycle hooks, watching endpoint changes, finite connection lifetimes, or a database-aware proxy?
4 Answers
A database-aware connection pooler is often the cleanest solution. The application pods connect to the pooler, and the pooler manages a smaller, shared set of database connections. Options include RDS Proxy for supported AWS databases, PgBouncer for PostgreSQL, and other PostgreSQL poolers. The pooler operates continuously rather than waiting for a scaling event, so it is less sensitive to how many application replicas are running.
The bigger scaling risk is often the total pool size. If every pod maintains a pool of 10 connections, increasing from 3 replicas to 20 raises the potential database connections from 30 to 200. That can exhaust the database before stale connections become a problem. Use a finite max lifetime for normal recycling, and size each per-pod pool based on the database's actual connection budget.
HPA only changes the replica count; it does not know anything about application connections or connection pools. The application has to handle this itself, usually by setting a finite connection lifetime or idle timeout so connections are recycled naturally. Watching endpoint changes and rotating the pool can also work, but it adds application complexity. A proxy can handle similar behavior when changing the application is difficult.
A load balancer or service mesh may distribute new connections across the available endpoints, but existing persistent connections usually are not moved automatically. Keepalive settings can limit how long they remain pinned, but they do not replace application-level pool recycling or database-side pooling.
A proxy is a reasonable option when the application cannot safely rebuild its own pool, though it is still important to configure connection and idle timeouts appropriately.