Hey everyone,
I'm currently running a Kong Ingress Controller on our AKS clusters, with three replicas and pod anti-affinity settings configured for preferred scheduling. We also have topology spread constraints with a MaxSkew of 1. Plus, we implemented a Pod Disruption Budget that ensures at least one instance stays available. Our setup consists of a minimum of 15 nodes, scaling up to 150-200 in production.
I'm wondering if it would be beneficial to explore using Horizontal Pod Autoscaler (HPA) instead of sticking with static replicas. While we've got HPA set up for our application workloads, we haven't done the same for platform components like Kong, Prometheus, and external DNS, among others.
Is it considered best practice to enable HPA for resources like these? I feel like it might add unnecessary complexity, but I'd love to hear your experiences if you've encountered similar situations.
1 Answer
For ingress controllers, I usually recommend running them as a daemon set. This way, you get an even load spread without any hotspots. The downside? It can be tough to trace requests if you’re not using centralized logging.
Is this approach only for bare-metal clusters? Don’t you end up with resource overhead if a pod runs on every node?