← Back to Blog

Upgrade the GPU Operator driver without taking every node down


One GPU node slid out on rails while the rest of the rack stays lit

I change driver.version in the ClusterPolicy. A minute later every GPU node is NotReady. The DaemonSet used the default rollout and restarted the driver on all of them together. Inference stops even though I only wanted a driver bump.

The problem

The driver pod is not a stateless API replica. While it restarts, the device plugin drops the GPU from allocatable. If that happens on every node in the same minute, the queue has nowhere to run. A surge of maxUnavailable on that DaemonSet is an outage.

The rollout I use

  1. I set the driver DaemonSet update strategy so only one pod is unavailable.
  2. I cordon one GPU node and drain it. Running pods move, or they wait if they cannot fit elsewhere. I would rather see that on one node than on all of them.
  3. I let the new driver pod reach Running. I run nvidia-smi on that node. I check allocatable is back.
  4. I uncordon, watch one real request land, then I take the next node.

I do not change the driver version and the operator version in the same change. If the new driver fails to compile, I still have the previous operator, and I can put the old version back on the canary node before I touch the rest. The first install is the setup note.

Commercial Engineering & Platform Infrastructure Advisory

Need hands-on senior engineering to audit, scale, or stabilize your infrastructure?

From bare-metal Kubernetes and immutable Talos Linux to drastically cutting cloud infrastructure bills, resolving PostgreSQL deadlocks, and architecting sovereign AI systems, I work directly with production engineering teams to deliver resilient foundations.