Kubernetes · troubleshooting

static pods may not reference

From Kubernetes 1.34 the kubelet refuses a static Pod that refers to an API object: a ConfigMap, Secret, ServiceAccount or PersistentVolumeClaim. Kubernetes 1.37 removed the feature gate that allowed opting out. If the refused manifest is a control-plane component, that component does not come back after the kubelet upgrade. Put the data the Pod needs on the node and mount it with hostPath, or run the workload as a normal Pod.

First published 20 Oct 2026

New to this? Read what is actually happening first, then come back to the steps.

On this page · 10 sections

Before you start

Shell access to the node: static Pod manifests and the kubelet's log are on the node, and the Pod may never appear in kubectl.
The kubelet's staticPodPath; with kubeadm it is /etc/kubernetes/manifests.
For control-plane nodes, an etcd backup before editing anything in the manifests directory.

Run this first

step 1 of 3

sudo journalctl -u kubelet --since "-30min" | grep -i "may not reference"

Look for static pods may not reference <kind>, and path="…" naming the manifest

The kubelet logs which file it refused and which kind of reference it found.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

Which kubelet version is the node running?

kubelet --version on the node

Cause space

2 of 2 still possible

A static Pod manifest uses a ConfigMap, Secret or service accountCommon
The cluster opted out in 1.34–1.36, and 1.37 removed the opt-outOccasional

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Everything else that changed from 1.34 to 1.37

Or diagnose it manually

In this order. The first command usually contains the whole answer.

Step 1
```
sudo journalctl -u kubelet --since "-30min" | grep -i "may not reference"
```
The kubelet logs which file it refused and which kind of reference it found.
Look for static pods may not reference <kind>, and path="…" naming the manifest
Step 2
```
sudo ls -la /etc/kubernetes/manifests/
```
The files the kubelet runs as static Pods. Use your kubelet's staticPodPath if it is not the kubeadm default.
Look for The manifest named in the log, and anything unexpected in the directory
Step 3
```
sudo crictl ps -a
```
Shows which control-plane containers are actually running on the node, independent of the API server.
Look for No container for the refused component

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

A static Pod manifest uses a ConfigMap, Secret or service account

Common

Manifests written before 1.34, or copied from a Deployment's Pod template, often take configuration from a ConfigMap or Secret, or set serviceAccountName. The kubelet now refuses all of these in a static Pod.

Confirm

sudo grep -nE 'serviceAccountName|secretKeyRef|configMapKeyRef|secretRef|configMapRef|configMap:|secret:|persistentVolumeClaim:|projected:|csi:|ephemeral:|resourceClaims' /etc/kubernetes/manifests/*.yaml

Any match in the file the kubelet log names

Fix

Put the configuration on the node as a file and mount it with a hostPath volume, the way kubeadm's own manifests mount certificates and kubeconfigs.
Or, if it does not need to start before the API server, run it as a DaemonSet or Deployment instead, where references are allowed.
Move the original file out of the manifests directory before editing, to a directory the kubelet does not read, so a half-edited file is never picked up.

The cluster opted out in 1.34–1.36, and 1.37 removed the opt-out

Occasional

Setting the PreventStaticPodAPIReferences feature gate to false kept such Pods running through 1.36. Kubernetes 1.37 removed the gate, so the same manifests are refused after the upgrade.

Confirm

sudo grep -rn PreventStaticPodAPIReferences /var/lib/kubelet/config.yaml /var/lib/kubelet/kubeadm-flags.env /etc/systemd/system/kubelet.service.d/ 2>/dev/null

The gate set to false in the kubelet's configuration or flags

Fix

Remove the gate from the kubelet configuration; a removed gate is no longer accepted.
Fix the manifests as for the first cause before upgrading the kubelet to 1.37.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

A static Pod is one the kubelet starts directly from a file on the node, without asking the API server. Control planes built with kubeadm run this way: etcd and the API server are static Pods. Because they start without the API server, they cannot depend on objects stored in it, such as ConfigMaps or Secrets. Kubernetes used to start such a Pod anyway and fail quietly; since 1.34 the kubelet refuses the file and logs why.

Words used on this page

Static Pod: A Pod the kubelet runs from a manifest file in its staticPodPath directory (or a URL), without the API server or a controller.
Mirror Pod: A read-only copy of a static Pod that the kubelet creates in the API server so kubectl can see it.
API object reference: Any part of a Pod spec that names another object in the API server. The kubelet's list covers service accounts, Secrets and ConfigMaps (in env or volumes), PersistentVolumeClaims, CSI and ephemeral volumes, projected volumes, cluster trust bundles, Pod certificates and resource claims.
PreventStaticPodAPIReferences: The feature gate that let clusters opt out of the 1.34 restriction. Kubernetes 1.37 removed it, so the restriction always applies.

What is actually happeningadvanced

The kubelet reads every file in its staticPodPath. Since 1.34 (#131837) it checks each Pod for references to API objects and, if it finds one, rejects the file with "static pods may not reference" followed by the kind, for example configmaps (via configmap volumes). Before, such Pods ran while the API server refused their mirror Pod, so they were silently running even though the mirror Pod could not be created.

The kubelet logs the rejection as "Could not process manifest file", with the error and the file's path as fields. The Pod is not started, so on a control-plane node the component it runs is simply missing after the upgrade.

1.34 let clusters opt out with the PreventStaticPodAPIReferences feature gate. The 1.37 release removed the gate, so an opt-out set during the 1.34–1.36 releases stops working at 1.37.

The kubelet's check covers: a serviceAccountName, Secrets and ConfigMaps (env references and volumes), PersistentVolumeClaims, CSI, ephemeral, azureFile and glusterfs volumes, and projected volumes with ConfigMaps, Secrets, service account tokens, cluster trust bundles or Pod certificates, and resource claims. The message names which one it found first.

How to tell this is your problemsignals

Where	What you see
journalctl -u kubelet	"Could not process manifest file" with err="static pods may not reference configmaps (via configmap volumes)" and the manifest's path
crictl ps on the node	No container for the component whose manifest was refused
kubectl get pods -n kube-system	The component's mirror Pod is missing, or the API server is unreachable if it was kube-apiserver or etcd

How to know it is actually fixed

sudo journalctl -u kubelet shows no new "may not reference" lines after the fix.
sudo crictl ps shows the component's container running.
kubectl get pods -n kube-system shows its mirror Pod again, and kubectl get nodes shows the node Ready.

Stopping it happening again

Before upgrading kubelets to 1.34 (and again to 1.37), run the grep above on every node's manifests directory.
Keep static Pods minimal: configuration as files on the node, nothing that lives in the API server.
Read each release's Urgent Upgrade Notes; this change is listed in 1.34's.

Security considerations

Moving a Secret's contents into a file on the node means protecting that file: root-owned and readable only by what needs it, as kubeadm does for the certificates under /etc/kubernetes/pki.
Back up etcd before changing control-plane manifests. A broken etcd or kube-apiserver manifest takes the whole cluster's API down until the file is fixed on the node.
Do not re-enable the old behaviour by staying on an old kubelet. The restriction exists because the Pod used to run while its mirror Pod failed, which hid the problem.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it