Upgrade the GPU Operator driver without taking every node down

I change driver.version in the ClusterPolicy. A minute later every GPU node is NotReady. The DaemonSet used the default rollout and restarted the driver on all of them together. Inference stops even though I only wanted a driver bump.
The problem
The driver pod is not a stateless API replica. While it restarts, the device plugin drops the GPU from allocatable. If that happens on every node in the same minute, the queue has nowhere to run. A surge of maxUnavailable on that DaemonSet is an outage.
The rollout I use
- I set the driver DaemonSet update strategy so only one pod is unavailable.
- I cordon one GPU node and drain it. Running pods move, or they wait if they cannot fit elsewhere. I would rather see that on one node than on all of them.
- I let the new driver pod reach Running. I run
nvidia-smion that node. I check allocatable is back. - I uncordon, watch one real request land, then I take the next node.
I do not change the driver version and the operator version in the same change. If the new driver fails to compile, I still have the previous operator, and I can put the old version back on the canary node before I touch the rest. The first install is the setup note.
Commercial Engineering & Platform Infrastructure Advisory
Need hands-on senior engineering to audit, scale, or stabilize your infrastructure?
From bare-metal Kubernetes and immutable Talos Linux to drastically cutting cloud infrastructure bills, resolving PostgreSQL deadlocks, and architecting sovereign AI systems, I work directly with production engineering teams to deliver resilient foundations.