GPU Operator driver pod CrashLoopBackOff after a kernel update

I get this one the morning after a node took an unattended upgrade. The GPU Operator driver pod is in CrashLoopBackOff. nvidia-smi is gone. Inference pods sit Pending. The log in the driver container says it cannot build the kernel module, and the missing piece is almost always linux-headers for the kernel that is running now, not the kernel that was running when the operator installed.
The problem
The operator compiles the NVIDIA kernel module inside a container. That compile needs the headers that match uname -r on the host. An upgrade moves the running kernel and leaves the headers behind. The pod retries, fails the same way, and the node reports zero nvidia.com/gpu.
What I check
uname -r
kubectl -n gpu-operator logs daemonset/nvidia-driver-daemonset --tail=80
ls /usr/src | grep linux-headers
If the headers package for that exact kernel is absent, stop there. Reinstalling the operator does not create headers.
The fix I leave on the node
- Install the matching headers:
sudo apt-get install -y linux-headers-$(uname -r). - Delete the driver pod on that node so the DaemonSet starts a fresh compile. Do not install
nvidia-driverfrom apt. The operator owns the driver. - Turn off unattended kernel upgrades on GPU nodes, or hold the kernel package. A GPU node is not a place for a surprise reboot into a new kernel.
- Cordon the node before the next planned kernel bump, install headers first, then let the driver pod rebuild, then uncordon.
The parent note on bringing the operator up cleanly is Provisioning the NVIDIA GPU Operator.
Related notes: Two drivers on one node, Upgrading the driver one node at a time.
Commercial Engineering & Platform Infrastructure Advisory
Need hands-on senior engineering to audit, scale, or stabilize your infrastructure?
From bare-metal Kubernetes and immutable Talos Linux to drastically cutting cloud infrastructure bills, resolving PostgreSQL deadlocks, and architecting sovereign AI systems, I work directly with production engineering teams to deliver resilient foundations.