← Back to Blog

GPU Operator driver pod CrashLoopBackOff after a kernel update


GPU server chassis open after a driver failure

I get this one the morning after a node took an unattended upgrade. The GPU Operator driver pod is in CrashLoopBackOff. nvidia-smi is gone. Inference pods sit Pending. The log in the driver container says it cannot build the kernel module, and the missing piece is almost always linux-headers for the kernel that is running now, not the kernel that was running when the operator installed.

The problem

The operator compiles the NVIDIA kernel module inside a container. That compile needs the headers that match uname -r on the host. An upgrade moves the running kernel and leaves the headers behind. The pod retries, fails the same way, and the node reports zero nvidia.com/gpu.

What I check

uname -r
kubectl -n gpu-operator logs daemonset/nvidia-driver-daemonset --tail=80
ls /usr/src | grep linux-headers

If the headers package for that exact kernel is absent, stop there. Reinstalling the operator does not create headers.

The fix I leave on the node

  1. Install the matching headers: sudo apt-get install -y linux-headers-$(uname -r).
  2. Delete the driver pod on that node so the DaemonSet starts a fresh compile. Do not install nvidia-driver from apt. The operator owns the driver.
  3. Turn off unattended kernel upgrades on GPU nodes, or hold the kernel package. A GPU node is not a place for a surprise reboot into a new kernel.
  4. Cordon the node before the next planned kernel bump, install headers first, then let the driver pod rebuild, then uncordon.

The parent note on bringing the operator up cleanly is Provisioning the NVIDIA GPU Operator.

Related notes: Two drivers on one node, Upgrading the driver one node at a time.

Commercial Engineering & Platform Infrastructure Advisory

Need hands-on senior engineering to audit, scale, or stabilize your infrastructure?

From bare-metal Kubernetes and immutable Talos Linux to drastically cutting cloud infrastructure bills, resolving PostgreSQL deadlocks, and architecting sovereign AI systems, I work directly with production engineering teams to deliver resilient foundations.