← Back to Blog

Cilium blocks the GPU Operator from pulling the driver


Network cables into a switch in front of a GPU server

The driver pod sits in ImagePullBackOff, or it starts and then cannot fetch the NVIDIA installer. Every other namespace pulls images fine. The difference is a Cilium policy that denies egress from gpu-operator.

The problem

A cluster-wide default deny is the right baseline. The operator namespace is not exempt. The driver container has to reach the registry and the NVIDIA download host. With egress closed, the pod retries and the node never advertises a GPU. People then disable the policy. That is the wrong fix.

What I open

I allow egress from the gpu-operator namespace to DNS, to the registry that holds nvcr.io mirrors, and to the HTTPS endpoint the driver uses. I also allow the device plugin to reach the kubelet. I do not allow the namespace to reach the rest of the cluster.

kubectl -n gpu-operator describe pod -l app=nvidia-driver-daemonset | grep -A5 Events
cilium monitor --related-to <driver-pod>

If the drop is a policy deny, I add the egress rule and delete the pod. If the drop is DNS, I fix DNS first. A registry allow rule does nothing when the pod cannot resolve the name.

The install order is still the one in the operator note. I do not turn Cilium off to make the pull work.

Commercial Engineering & Platform Infrastructure Advisory

Need hands-on senior engineering to audit, scale, or stabilize your infrastructure?

From bare-metal Kubernetes and immutable Talos Linux to drastically cutting cloud infrastructure bills, resolving PostgreSQL deadlocks, and architecting sovereign AI systems, I work directly with production engineering teams to deliver resilient foundations.