← Back to Blog

GPU Operator shows zero GPUs after a node reboot


GPU node powering back on in a dark rack

After a reboot the node is Ready, and a describe still shows nvidia.com/gpu: 0. Sometimes nvidia-smi already works on the host. The scheduler does not care about the host command. It only schedules on the allocatable count the device plugin published.

The problem

The toolkit pod and the device-plugin pod race the driver pod. If they start first, they register nothing. They do not always retry once the driver finishes. The node looks healthy and has no GPU.

The order I want

kubectl -n gpu-operator get pods -o wide | grep <node>

Driver pod Running first. Then toolkit. Then device plugin. If the driver is Running and the device plugin is older than the driver, I delete only the device-plugin pod on that node.

kubectl -n gpu-operator delete pod -l app=nvidia-device-plugin-daemonset --field-selector spec.nodeName=<node>

I wait for it to come back, then I check allocatable again:

kubectl get node <node> -o jsonpath='{.status.allocatable.nvidia.com/gpu}{"\n"}'

If that stays at zero, the driver did not actually load. I go to the driver log instead of restarting the plugin a second time. Restarting the plugin cannot invent a device the driver failed to create.

The install order is the one in the operator setup.

Commercial Engineering & Platform Infrastructure Advisory

Need hands-on senior engineering to audit, scale, or stabilize your infrastructure?

From bare-metal Kubernetes and immutable Talos Linux to drastically cutting cloud infrastructure bills, resolving PostgreSQL deadlocks, and architecting sovereign AI systems, I work directly with production engineering teams to deliver resilient foundations.