Hey everyone! I've got a Kubernetes cluster with 16 worker nodes, and most of my services are running as a daemonset to help with load distribution. Right now, I have more than 75 pods on each node. I'm wondering if increasing the number of pods on the worker nodes will negatively affect CPU performance due to the high number of context switches. What do you all think?
6 Answers
Yeah, using a daemonset for load distribution isn't really how it’s meant to be used! Switching to deployments can change your approach to distribution. You might be creating too many unnecessary pods for smaller apps and not enough for larger ones. Though your concerns are valid, you can reduce context switch overhead by trying fewer pods with more resources.
Using daemonsets for load/performance balancing is not a good idea! Just switch to deployments for better results.
Yes, context switching can affect performance, but the impact is usually small. If optimization is a concern, look into the Kubernetes CPU Manager and NUMA awareness. This is especially important for workloads that are very sensitive to latency.
Context switches are typically a result of your workload rather than just the number of processes. I wouldn’t worry too much unless you start noticing issues. In general, a deployment should be fine for scaling applications, and you might consider HPA for spikes. And don’t forget PDBs to keep some pods running at all times!
CPUs are generally pretty quick, so monitoring your CPU load will help determine if you need more cores. However, the way you’re using a daemonset might be your main concern here—running 16 replicas of everything seems excessive! Instead, consider running your services as deployments with a more reasonable number of replicas. This approach can help save capacity and allocate resources better to CPU-heavy workloads. Also, look into pod disruption budgets; they make draining nodes for updates way easier!
Deployment versus daemonset doesn't matter without proper measurement of what scale you actually need. The same situation could happen with anti-affinity on deployments! Having multiple pods on one node is fine for uptime and rollouts, but definitely get a couple of nodes for high availability and performance.
Don’t forget about pod affinity rules! They can help spread workloads across nodes more effectively.