Skip to main content
DevOps Insightsguides & answers

Kubernetes · troubleshooting

failed calling webhook

The API server could not reach an admission webhook that has to approve your request, so it rejected the request. The text after failed to call webhook says why: no pods behind the service, a timeout, or a certificate. Fix that webhook, or remove its configuration if the software is gone.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 10 sections

Before you start

  • The full error, which names the webhook and the URL the API server tried.
  • kubectl access with permission to list webhook configurations, services and pods in the webhook's namespace.

Run this first

step 1 of 2
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

Look for The configuration that contains the webhook name from the message

Lists every webhook the API server may call; find the one named in the error.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

What does the error say after failed to call webhook?

Cause space

4 of 4 still possible

  • Nothing is running behind the webhook's serviceCommon
  • The software was uninstalled but its webhook configuration was left behindCommon
  • The API server cannot reach the webhook pods in timeCommon
  • The API server does not trust the webhook's certificateOccasional

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

    Lists every webhook the API server may call; find the one named in the error.

    Look for The configuration that contains the webhook name from the message

  2. Step 2

    kubectl get pods,svc,endpointslices -n NAMESPACE

    Shows whether the webhook's service exists and has ready endpoints.

    Look for Running, Ready pods and an EndpointSlice with addresses

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

Nothing is running behind the webhook's service

Common

The webhook's pods are crash-looping, scaled to zero, unscheduled, or not ready yet after an install or a node failure. The API server finds the service but no endpoint to send the request to.

Confirm

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get pods,endpointslices -n NAMESPACE

no endpoints available for service in the error, and the webhook's pods not Running and Ready

Fix

  • Fix the webhook's pods first: read why they are not ready (kubectl describe pod, kubectl logs). A webhook is usually down for the same reasons any pod is.
  • If the webhook blocks its own repair, a cluster admin can temporarily set that webhook's failurePolicy to Ignore, fix the pods, then set it back.

The software was uninstalled but its webhook configuration was left behind

Common

An ingress controller, cert-manager or a policy tool was removed (namespace deleted, Helm release half-uninstalled) while its cluster-scoped webhook configuration stayed. Every matching request now calls a service that no longer exists.

Confirm

kubectl get validatingwebhookconfigurations -o custom-columns=NAME:.metadata.name,SVC:.webhooks[*].clientConfig.service.name,NS:.webhooks[*].clientConfig.service.namespace
kubectl get ns NAMESPACE

A configuration pointing at a namespace or service that no longer exists (services "NAME" not found)

Fix

  • Once you are sure the software is gone, delete its configuration: kubectl delete validatingwebhookconfiguration NAME (and the mutating one, if any).
  • If you still use the software, reinstall it properly instead, so the configuration and the service match again.
kubectl delete validatingwebhookconfiguration NAME

Only after confirming the owning software is really removed.

The API server cannot reach the webhook pods in time

Common

The control plane runs outside your nodes (managed clusters) and a firewall, security group or network policy blocks it from the webhook pod's port, or the webhook answers too slowly. The call times out after the webhook's timeoutSeconds.

Confirm

kubectl get validatingwebhookconfiguration NAME -o jsonpath='{.webhooks[*].timeoutSeconds}{"\n"}'
kubectl get svc SERVICE -n NAMESPACE -o jsonpath='{.spec.ports[*].targetPort}{"\n"}'

context deadline exceeded or i/o timeout in the error, while the pods are Running

Fix

  • Allow traffic from the control plane to the webhook pods on their target port (the container port, not the service port 443). On managed clusters this is usually a node security group or firewall rule.
  • Check NetworkPolicies in the webhook's namespace: a default-deny policy blocks the API server too.

The API server does not trust the webhook's certificate

Occasional

The webhook's serving certificate was renewed or regenerated, but the caBundle in the webhook configuration still holds the old CA, or the certificate does not cover the service's DNS name.

Confirm

kubectl get validatingwebhookconfiguration NAME -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -subject -enddate

x509: certificate signed by unknown authority or certificate is valid for … not … in the error, or an expired CA

Fix

  • Let the tool that manages the certificate re-inject the caBundle (for example cert-manager's CA injector, or the chart's certificate job), or reinstall the release that owns the webhook.
  • Make sure the certificate's names include SERVICE.NAMESPACE.svc, the name the API server calls.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Some tools install a gatekeeper into your cluster: every time you create or change certain objects, the API server first asks that tool's small web service whether it is allowed. If the gatekeeper is not running, is unreachable or has the wrong certificate, the API server cannot ask, and by default it says no to be safe. So a kubectl apply fails with an error that names a webhook you may never have heard of.

Words used on this page

Admission webhook
An HTTPS service the API server calls to validate or change objects before saving them. Installed by tools such as ingress controllers, cert-manager or policy engines.
ValidatingWebhookConfiguration / MutatingWebhookConfiguration
The cluster objects that tell the API server which webhooks to call, for which requests, and how.
failurePolicy
What to do when the webhook cannot be called: Fail (the default) rejects the request, Ignore lets it through.
caBundle
The CA certificate the API server uses to trust the webhook's TLS certificate.
What is actually happening

Before saving an object that a webhook's rules match, the API server sends it to that webhook over HTTPS. If the call fails and failurePolicy is Fail, the default, the request is rejected as Internal error occurred: failed calling webhook "NAME": failed to call webhook: …. With Ignore, the request goes ahead and the failure is only logged.

The call can fail for network errors, timeouts or connection failures, a non-2xx or malformed answer, or the API server failing to build the request. The default timeout is 10 seconds; a webhook may set 1 to 30.

Everything after failed to call webhook is the actual cause, in the API server's words: no endpoints available for service, services "NAME" not found, context deadline exceeded, connection refused, or an x509 error.

Because the webhook runs as pods in the cluster, it can block the very changes needed to repair it, for example when its own Deployment matches its rules.

How to tell this is your problem
WhereWhat you see
kubectl apply or helm upgradeError from server (InternalError): Internal error occurred: failed calling webhook "NAME": failed to call webhook: Post "https://SERVICE.NAMESPACE.svc:443/PATH?timeout=10s": …
End of the same messageno endpoints available for service "SERVICE", or context deadline exceeded, or x509: certificate signed by unknown authority

How to know it is actually fixed

  • The kubectl apply or helm upgrade that failed now succeeds.
  • Any webhook you set to Ignore while repairing is back to Fail.

Stopping it happening again

  • Run webhooks with more than one replica and a PodDisruptionBudget, so a single node failure does not block the API.
  • Exclude the webhook's own namespace and kube-system with a namespaceSelector, so it cannot block its own repair or core components.
  • Uninstall tools with their own uninstaller or helm uninstall, then check that no webhook configuration is left.

Security considerations

  • Do not switch a security webhook (policy, image signature, secret injection) to failurePolicy: Ignore to get unstuck: while it is down, anything it would have blocked gets in.
  • Delete a webhook configuration only when the software that owned it has really been removed. Deleting the configuration of a running policy engine switches the policy off.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next