What should you check before upgrading a Kubernetes cluster to 1.37?

0
0
Asked By MellowCedar42 On

Kubernetes 1.37 includes several changes that may cause components to reject their configuration, stop serving APIs, or silently affect monitoring after an upgrade. The most important checks include removing feature gates that were removed or locked, cleaning obsolete kubelet cAdvisor flags from systemd and bootstrap configuration, migrating kubeadm v1beta3 configuration with an older binary before upgrading, and checking whether static Pods reference Secrets or ConfigMaps.

Clusters using the alpha scheduling.k8s.io/v1alpha2 Workload or PodGroup APIs need special care: delete those stored objects before upgrading and recreate them as v1beta1 after the control plane reaches 1.37. This does not apply to Kueue's separate Workload CRD. Also note that eventRecordQPS: 0 now means unlimited instead of falling back to the client-go default, and several API-server, DRA, and cAdvisor metrics have been renamed or removed without causing PrometheusRules to report errors.

The upgrade also warns when kube-proxy's Linux mode is implicit. Set the mode explicitly now, especially if you use IPVS, since the default is expected to move to nftables in a future release and IPVS is being phased out. Finally, SELinux mount-time labeling may expose conflicting labels on shared volumes and leave Pods stuck in ContainerCreating.

Useful checks include inspecting each node's effective kubelet configuration through configz, searching PrometheusRules for removed cAdvisor series, and checking the kube-proxy ConfigMap for an explicit mode. Test the upgrade in a non-production environment first, then repeat the checks in production. A general cluster-hygiene tool can help identify probes, limits, RBAC, and deprecated APIs, but it will not detect release-specific issues such as removed flags, feature gates, eventRecordQPS changes, or the scheduling API transition.

3 Answers

Answered By QuartzHawk7 On

The feature gates and scheduling API transition are probably the biggest risks for batch or gang-scheduling users. Remove gates such as JobSuccessPolicy and any other removed or locked gates from component configuration before upgrading, because the 1.37 binary can refuse to start if they are still present.

The scheduling.k8s.io/v1alpha2 Workload and PodGroup APIs are not an in-place conversion. Delete stored v1alpha2 objects before the control-plane upgrade, then recreate them as v1beta1 afterward. This is separate from Kueue's Workload CRD, which uses a different API group.

Answered By NorthwindMica19 On

Do a node-by-node search of systemd units, cloud-init, and bootstrap scripts for the removed kubelet cAdvisor flags. Nothing inside the cluster will automatically catch those settings, and a node can fail to start its kubelet only when it gets restarted with the new binary.

I would also migrate any kubeadm v1beta3 configuration with a pre-1.37 kubeadm binary before beginning the upgrade, since the newer version no longer supports that migration.

Answered By MellowCedar42 On

A general hygiene scanner is useful for finding bad probes, missing resource settings, RBAC problems, and deprecated APIs, so it is worth running in staging and production. However, it is not release-aware enough to find the 1.37-specific problems here. Pair it with direct checks for feature gates, kubelet flags, eventRecordQPS, removed metrics, kube-proxy mode, and stored scheduling v1alpha2 objects.

The eventRecordQPS change is especially easy to miss: an explicit zero now means unlimited event recording. Set a deliberate value, such as 50, unless unlimited event traffic is intentional.

CopperVale5 -

The safest order is still staging first, fix everything found there, repeat the checks in production, and only then roll out the new control-plane and node binaries. Configuration errors in removed gates or flags may not appear until a component restarts.

Related Questions

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.