← Back to Blog

Two NVIDIA drivers on one node: the operator and a host package


Two server boards side by side, each with a GPU

The symptom is messy. nvidia-smi works over SSH. The pod starts and dies with Failed to initialize NVML: Driver/library version mismatch. Two drivers are on the machine. They are not the same build.

The problem

Someone installed nvidia-driver with apt so the card would show up before the operator existed. Later the GPU Operator also deploys a driver container. The kernel module and the userspace library come from different builds. The host tool and the container do not agree.

The fix I actually use

I pick an owner. On clusters I run, the operator owns the driver.

  1. Cordon and drain the node.
  2. Purge the host packages: sudo apt-get purge -y 'nvidia-*' 'libnvidia-*'. Then reboot.
  3. Confirm the operator driver pod is the thing that loads the module. lsmod | grep nvidia should appear only after that pod is Running.
  4. If the host driver must stay, I do the opposite in the ClusterPolicy: driver.enabled: false. The operator then deploys the toolkit and the device plugin, and it stops trying to load a second module.

I do not leave both enabled to "see which one wins". The mismatch comes back on the next reboot. The clean install path is in the operator note.

Commercial Engineering & Platform Infrastructure Advisory

Need hands-on senior engineering to audit, scale, or stabilize your infrastructure?

From bare-metal Kubernetes and immutable Talos Linux to drastically cutting cloud infrastructure bills, resolving PostgreSQL deadlocks, and architecting sovereign AI systems, I work directly with production engineering teams to deliver resilient foundations.