I recently deployed StarRocks DB in Kubernetes using Karpenter's node pools, which by default cause nodes to expire after 30 days. I suspect that there was a missing Pod Disruption Budget (PDB) when I set up my operator for the database. I'm looking for advice on how to keep my databases available with Karpenter's node pools, especially when all the nodes are set to expire at the same time. Also, please don't suggest the 'do-not-disrupt' annotation, as it won't help with removing the old nodes, and Karpenter will just spin up new ones.
3 Answers
Honestly, I think Karpenter just isn't reliable for databases. I talked with the Karpenter developers, and they mentioned there are no fixes planned for this issue. Creating a PDB won't protect you from losing all replicas at once due to Karpenter's expiration behavior.
You should definitely add a Pod Disruption Budget (PDB). If your operator doesn't support it out of the box, consider using Kyverno to create a policy that generates PDBs for your database pods. Setting up a custom node pool for your DB pods, with tuned TTL and consolidation settings, might help reduce disruption. Since nodes will roll over inevitably, it would be wise to explore high availability options too if disruptions are a concern.
In my experience, having PDBs, health checks, and more than one replica is essential for a smooth replacement of Karpenter nodes.
Expiration ignores PDB. Since v1. Sorry.
Expiration ignores PDB. Since v1. Sorry.