Kubernetes upgrades, 1.34 to 1.37: what breaks

What you see

static pods may not reference %s

From kubelet source, pkg/kubelet/config/common.go at v1.37.0; %s is the kind, for example configmaps (via configmap volumes).

Check before upgrading

sudo grep -nE 'serviceAccountName|secretKeyRef|configMapKeyRef|secretRef|configMapRef|configMap:|secret:|persistentVolumeClaim:|projected:|csi:|ephemeral:|resourceClaims' /etc/kubernetes/manifests/*.yaml

Any match. These are the references the kubelet checks: service accounts, Secrets and ConfigMaps (env or volumes), PVCs, CSI and ephemeral volumes, projected volumes and resource claims. Use your kubelet's staticPodPath if it is not the kubeadm default.

Fix

• Move the referenced data onto the node and mount it with hostPath, as kubeadm's own manifests do for certificates and kubeconfigs.
• Or run the workload as a normal Pod, DaemonSet or Deployment, which may reference API objects.

What you see

failed to parse kubelet flag: %w

From kubelet source, cmd/kubelet/app/server.go at v1.35.0, wrapping pflag's "unknown flag: --%s". Expected combined output, not captured: failed to parse kubelet flag: unknown flag: --cloud-config.

Check before upgrading

ps -o args= -C kubelet | tr ' ' '\n' | grep -- '--cloud-config'

Any output: the running kubelet has the flag. Remove it from the unit file, drop-in or env file it came from before upgrading.

Fix

• Delete the flag. Cloud integration belongs to the external cloud-controller-manager.

What you see

kubelet is configured to not run on a host using cgroup v1. cgroup v1 support is unsupported and will be removed in a future release

From kubelet source, pkg/kubelet/apis/config/validation/validation_linux.go at v1.35.0.

Check before upgrading

stat -fc %T /sys/fs/cgroup/

cgroup2fs means cgroup v2 and nothing to do. tmpfs means cgroup v1: this node will not run a 1.35 kubelet as configured.

Fix

• Move the node to cgroup v2. This is the fix that lasts, because cgroup v1 support is being removed.
• To keep cgroup v1 for now, set failCgroupV1: false in the kubelet configuration.
• With kubeadm, the 1.35 changelog says to do two things: edit the kube-system/kubelet-config ConfigMap and add failCgroupV1: false before upgrading, and ignore the SystemVerification preflight error.

What you see

failed to parse kubelet flag: %w

From kubelet source, cmd/kubelet/app/server.go at v1.35.0, wrapping pflag's "unknown flag: --%s". Expected combined output, not captured: failed to parse kubelet flag: unknown flag: --pod-infra-container-image.

Check before upgrading

sudo grep -rn -- '--pod-infra-container-image' /var/lib/kubelet/kubeadm-flags.env /etc/default/kubelet /etc/sysconfig/kubelet /etc/systemd/system/kubelet.service.d/ 2>/dev/null

Any match is a kubelet that will not start on 1.35. Also check kubeletExtraArgs in the kubeadm configuration you upgrade with.

Fix

• Remove the flag from every place it is set, then upgrade the kubelet. The sandbox image is configured in the container runtime now.

What you see

The ipvs proxier has been deprecated and will be disabled by default in Kubernetes 1.40 and removed in Kubernetes 1.43. Migrate to the 'nftables' proxier instead.

From kube-proxy source, cmd/kube-proxy/app/server_linux.go at v1.37.0 (1.35 and 1.36 print an earlier wording).

Check before upgrading

kubectl -n kube-system get configmap kube-proxy -o yaml | grep -n 'mode:'

mode: ipvs (kubeadm-managed kube-proxy). An empty mode means the default, which is not ipvs.

Fix

• Plan the move to mode: nftables and test it on one cluster first. The 1.35 release post links KEP-5495 for the reasoning.

What you see

CRI implementation should be updated to support RuntimeConfig. Falling back to using cgroupDriver from kubelet config.

From kubelet source, pkg/kubelet/kubelet.go at v1.36.0 and v1.37.0, logged at info level. The kubelet metric kubelet_cri_losing_support also reports the version in which your runtime loses support..

Check before upgrading

kubectl get nodes -o wide

The CONTAINER-RUNTIME column. containerd://1.7.x means the warning above applies; plan containerd 2.x before 1.38.

Fix

• Upgrade containerd to 2.x on your own schedule, before 1.38, and re-check the 1.38 notes when they are published.

What you see

git-repo volume plugin has been disabled; if necessary, it may be re-enabled by enabling the feature-gate `GitRepoVolumeDriver`

From kubelet source, pkg/volume/git_repo/git_repo.go at v1.36.0. On 1.36 and later the advice in it cannot be followed..

Check before upgrading

kubectl get pods -A -o json | jq -r '.items[] | select(any(.spec.volumes[]?; has("gitRepo"))) | "\(.metadata.namespace)/\(.metadata.name)"'

Any Pod listed (needs jq). Check the templates of Deployments, StatefulSets and CronJobs too, since a scaled-to-zero workload has no Pods to find.

Fix

• Do what the docs suggest: mount an emptyDir into an init container that clones the repository with git, then mount the same emptyDir into the app container.
• To stop new ones, the docs give this CEL expression for a ValidatingAdmissionPolicy: !has(object.spec.volumes) || !object.spec.volumes.exists(v, has(v.gitRepo))

Check before upgrading

grep -rn 'volume_operation_total_errors' <your dashboards and alert rules>

Any use of the old name. Replace it with volume_operation_errors_total.

Fix

• Rename the metric in queries, recording rules and alerts, and check that the alert can still fire.

What you see

spec.externalIPs is deprecated and may no longer be implemented in some clusters

From API server source, pkg/api/service/warnings.go at v1.36.0.

Check before upgrading

kubectl get svc -A -o jsonpath='{range .items[?(@.spec.externalIPs)]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'

Any Service listed.

Fix

• Move those Services to type LoadBalancer or a Gateway, as the deprecation post describes.

Check before upgrading

ps -o args= -C kubelet | tr ' ' '\n' | grep -E -- '--(application-metrics-count-limit|boot-id-file|container-hints|containerd|containerd-namespace|enable-load-reader|event-storage-age-limit|event-storage-event-limit|global-housekeeping-interval|log-cadvisor-usage|machine-id-file|storage-driver-)'

Any output. Remove the flag where it is set before upgrading the kubelet.

Fix

• Delete the flags. They have done nothing useful for several releases.

What you see

conflicting SELinux labels of volume %s: %q and %q

From kubelet source, pkg/kubelet/volumemanager/cache/desired_state_of_world.go at v1.37.0.

Check before upgrading

--controllers=*,selinux-warning-controller

Add this to kube-controller-manager's flags while still on 1.36. Then read the metric selinux_warning_controller_selinux_volume_conflict: it lists every conflicting pair. The kubelet metric volume_manager_selinux_volume_context_mismatch_warnings_total counts Pods that would really fail.

Fix

• For each conflicting workload, set spec.securityContext.seLinuxChangePolicy: Recursive to keep the old behaviour, or remove the conflict.
• The official post walks through this path step by step; follow it on 1.36 before upgrading.

Check before upgrading

sudo grep -n 'eventRecordQPS' /var/lib/kubelet/config.yaml

eventRecordQPS: 0. The changelog suggests an explicit value such as 50 if you want the previous behaviour.

Fix

• Set an explicit non-zero value, for example 50, before upgrading.

Check before upgrading

kubectl -n kube-system get deploy kube-dns coredns

A Deployment named kube-dns means you are on kube-dns. A Service named kube-dns proves nothing: kubeadm keeps that name for CoreDNS too.

Fix

• Migrate to CoreDNS.

Check before upgrading

kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx

Any Pod listed (the command the retirement announcement gives; run it with cluster-admin rights).

Fix

• Migrate to Gateway API or another Ingress controller. Read the official "Before You Migrate" post first: its five behaviours are where migrated routes start returning 404s or redirecting.