Linux / shell

CreateContainerConfigError from a missing Secret or ConfigMap

Written and reviewed by Sahil Srivastav

KubernetesConfiguration referencesStartup

CreateContainerConfigError

What this error actually means

Kubelet cannot construct the configuration needed to start a container. A frequent cause is an environment variable whose secretKeyRef or configMapKeyRef points to a missing object or key. The process has not reached application initialisation, so changing its log level or restarting its dependency will not repair this reference.

There are three identities to compare: namespace, object name and key name. Matching only the visible Secret name is insufficient. A Secret in staging cannot satisfy a pod in production, and an object containing DATABASE_URL does not satisfy a reference to DB_URL. Both mistakes can produce a waiting container even though an apparently relevant object exists.

Configuration-volume failures may appear through FailedMount events and ContainerCreating instead. Use the detailed event to identify the failing path rather than expecting one status string for every missing configuration object. CreateContainerConfigError is a category with other possible causes, including security-context validation.

Causes, most common first

1A required object was never created in this namespace. The deployment assumes configuration provisioning has already happened. An external secret controller may still be reconciling, or its provider access may have failed. The desired external secret resource is not the same thing as the generated Kubernetes Secret the pod consumes.
2The object exists but a required key does not. A credentials migration changes key spelling or case while the pod retains the earlier reference. Environment-variable names and data keys need not be identical, so compare the source key explicitly rather than scanning only the variable visible to the application.
3Rendered resource names disagree. A Helm release prefix, Kustomize name suffix or namespace overlay changes one side of the reference. Looking at an unrendered values file can conceal the mismatch that is obvious in the admitted pod and actual resource list.
4Rotation removed configuration still referenced by old replicas. An immutable Secret or ConfigMap is replaced under a new name and the previous object is deleted before all consumers move. Existing processes may keep their environment while replacement containers fail, creating a misleading partial outage.

When you see it

The image is available but the container has no application logs
Events name a Secret, ConfigMap or key that kubelet cannot resolve
A deployment succeeds in one namespace and stalls in another
A renamed or hash-suffixed configuration object exists, but pods reference the old name

How to diagnose it

Step 1

Read the kubelet’s exact unresolved reference

Substitute the affected namespace and pod for demo and app-pod. The event should name the object or key. If it instead names runAsNonRoot or another validation rule, follow that evidence rather than creating unrelated Secrets.

kubectl -n demo describe pod app-pod

Step 2

Inspect reference fields without exposing values

This prints valueFrom and envFrom references, not literal environment values or Secret contents. Also inspect init containers when they are the ones waiting. Compare object names with the event rather than guessing from the deployment name.

kubectl -n demo get pod app-pod -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{range .env[*]}{.name}{"\t"}{.valueFrom}{"\n"}{end}{.envFrom}{"\n"}{end}'

Step 3

List key names, not credential values

Replace app-config with the exact name from the failed reference. The Go template enumerates keys only. A NotFound response establishes an object or namespace mismatch; a present object without the required key establishes a schema mismatch.

kubectl -n demo get secret app-config -o go-template='{{range $key, $value := .data}}{{printf "%s\n" $key}}{{end}}'
kubectl -n demo get configmap app-config -o go-template='{{range $key, $value := .data}}{{printf "%s\n" $key}}{{end}}'

Step 4

Trace the configuration producer

If a controller or deployment pipeline creates the missing resource, inspect its status and events. Determine whether creation is still pending, provider permission failed, or the generated name differs. A user lacking permission to inspect Secrets does not by itself prove that kubelet cannot resolve the pod’s reference.

The fix

Restore the intended object and key through the approved configuration or secret-management workflow. Correct the pod reference when the existing object is authoritative; correct the producer when the rendered pod contract is authoritative. Preserve the namespace boundary rather than copying unrelated environment credentials to make startup succeed.

Make deployment ordering explicit: establish and verify required configuration before rolling out consumers. With asynchronously generated Secrets, wait for the controller’s successful reconciliation and the output object rather than assuming that applying its custom resource means the Secret already exists.

Do not mark a required credential optional just to clear the waiting state. That trades a precise configuration error for an application crash, a fallback credential or an unintended unauthenticated mode. Optional is appropriate only when the application implements and tests a legitimate absent-value behaviour.

Keep old named configuration available while active revisions still reference it. When updating environment-based configuration on already running containers, perform a controlled restart or rollout so new processes receive the value. Waiting containers can retry automatically once their required reference becomes available.

How to stop it coming back

Validate rendered namespace, object and key references before advancing a rollout
Treat Secret and ConfigMap key names as a versioned application contract
Gate consumer deployment on generated-secret readiness, not only resource submission
Retain configuration for supported rollback revisions and remove it deliberately

Practise production debugging in a real repository

Reading about a failure and reproducing one are different skills. Gronex ships broken backend repositories with failing test suites that encode the real invariant, so you debug from evidence instead of memorising symptoms.

FAQ

Why does kubectl logs return nothing?

The container configuration could not be assembled, so the application process has not run. Pod events and the referenced resources are the evidence sources at this stage.

Can a pod reference a Secret in another namespace?

An ordinary Secret reference in a pod resolves within that pod’s namespace. Provision the intended secret through the environment’s management workflow in the consumer namespace; the same name elsewhere does not satisfy it.

Does changing the Secret update an existing environment variable?

No. A running process retains the environment it received at startup. Mounted configuration has different update behaviour, and applications still need a deliberate reload strategy. Decide the delivery contract instead of treating the two forms as interchangeable.

runAsNonRoot blocks container creation Registry credentials and ImagePullBackOff Application starts then crash-loops

Other errors engineers hit next to this one

Full error and symptom index →