CreateContainerConfigError

Before you start

• kubectl access to the pod's namespace, including events and the ConfigMaps or Secrets the pod uses.
• The pod's manifest, or the template of the Deployment or StatefulSet that owns it.
• For the runAsNonRoot messages, a way to read the image's configured user: docker image inspect or crane config.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Or diagnose it manually

In this order. The first command usually contains the whole answer.

1. Step 1

   kubectl get events -n <namespace> --field-selector involvedObject.name=<pod>,reason=Failed

   The event message is the only place, besides the status field, that says what went wrong. kubectl describe's State section does not show it.

   Look for Error: followed by not found, couldn't find key, runAsNonRoot, or a mount message

2. Step 2

   kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses[*].state.waiting.message}'

   The same text without the Error: prefix, and still there after the events have expired. Use .status.initContainerStatuses for an init container.

   Look for The name of the missing object or key, or the user check that failed

3. Step 3

   kubectl get configmap,secret -n <namespace>

   Lists what actually exists in the pod's namespace, which is the only namespace the pod can use.

   Look for The referenced name, spelled exactly as in the manifest

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

kubectl get events -n <namespace> --field-selector involvedObject.name=<pod>,reason=Failed
kubectl get configmap,secret -n <namespace>

kubectl describe configmap <name> -n <namespace>
kubectl describe secret <name> -n <namespace>

kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses[*].state.waiting.message}'
docker image inspect --format '{{.Config.User}}' <image>

securityContext:
  runAsNonRoot: true
  runAsUser: 10001

kubectl get events -n <namespace> --field-selector involvedObject.name=<pod>,reason=Failed

How to know it is actually fixed

• ✓kubectl get pod shows Running, or the replacement pods do after a rollout.
• ✓No new Failed events for the pod in the last minute.
• ✓kubectl exec <pod> -- env shows the expected variables, without printing Secret values anywhere you would not paste a password.

Stopping it happening again

• Deploy ConfigMaps and Secrets in the same release as the workloads that use them, so a reference cannot get ahead of its object.
• Add a CI check that every configMapKeyRef and secretKeyRef names an object and key the release creates. A server-side dry run does not catch this, because the kubelet resolves those references only when it starts the container.
• Build images with a numeric USER, so runAsNonRoot can be checked wherever the image runs.

Security considerations

• When checking a Secret, use kubectl describe secret, which lists keys with byte counts only. kubectl get secret -o jsonpath='{.data}' prints the values, base64 encoded, which is as good as plain text.
• Do not make a reference optional just to get the pod running. The container then starts without that setting, and an application that falls back to a default, an empty password or a disabled check is worse than one that does not start.
• Fix a runAsNonRoot failure by running as a non-zero user, not by removing runAsNonRoot. The Restricted Pod Security Standard requires it, and it is what stops a container compromise from starting as root.
• No RBAC change for the pod's service account fixes this. The kubelet reads ConfigMaps and Secrets for pods on its node through the Node authorizer, not with the pod's identity.