How do I change the network in my on-prem Kubernetes cluster?
Asked by
TechyNerd88
Hey everyone,
I'm currently working with a client who has an on-prem Kubernetes cluster set up with kubeadm. Their existing network CIDR is too small (10.0.0.0/28), but they have the option to expand to a larger network (10.0.1.0/24) through their cloud provider.
Has anyone managed to change the cluster network for the nodes? I'm drafting a workflow, but I want to make sure I'm not missing anything crucial:
- On worker nodes, I'll update the kubelet's listen address in `/etc/default/kubelet` by setting `KUBELET_EXTRA_ARGS='--node-ip «new ip»'`.
- For control plane access, we use an entry in `/etc/hosts`, so we'll need to point that to the new network's load balancer.
- On master nodes, I'll:
- Change the new IP in `/etc/kubernetes/manifests/etcd.yaml` for `etcd.advertise-client-url`, `advertise-client-urls`, `initial-advertise-peer-urls`, `initial-cluster`, `listen-client-urls`, and `listen-peer-urls`.
- Update the new IP in `/etc/kubernetes/manifests/kube-apiserver.yaml` for `kube-apiserver.advertise-address.endpoint`, `advertise-address`, and readiness/liveness probes.
- Modify `/etc/kubernetes/controller-manager.conf` and `/etc/kubernetes/scheduler.conf`.
Am I missing anything important? Thanks!
Answers
Are you looking to expand the cluster CIDR or just change the IPs of your nodes? It sounds like you're wanting to expand the CIDR to a new address since you can't modify the existing one.
Reply to this answer
Based on my understanding, simply changing some configurations isn’t enough. Kubeadm includes node IPs directly in certs and static Pod manifests. You’ll likely need to:
1. Back up etcd and tear down the static etcd Pods.
2. Update cluster member IPs and then restore your snapshot.
Also, if you’re using CNI plugins like Calico or Flannel, any node IP changes might break overlays. It can get complicated quickly, so you might want to update your processes carefully.
Reply to this answer
That sounds like a tough situation! Honestly, it might be easier to migrate your cluster with a tool like Velero. But I get that open source options can have restrictions depending on your organization.
Reply to this answer
I’m not entirely sure about all the ins and outs, but can you clarify how your cloud provider’s networking plays into your on-prem setup? Seems a bit off at first glance.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions