Two NVIDIA drivers on one node: the operator and a host package

The symptom is messy. nvidia-smi works over SSH. The pod starts and dies with Failed to initialize NVML: Driver/library version mismatch. Two drivers are on the machine. They are not the same build.
The problem
Someone installed nvidia-driver with apt so the card would show up before the operator existed. Later the GPU Operator also deploys a driver container. The kernel module and the userspace library come from different builds. The host tool and the container do not agree.
The fix I actually use
I pick an owner. On clusters I run, the operator owns the driver.
- Cordon and drain the node.
- Purge the host packages:
sudo apt-get purge -y 'nvidia-*' 'libnvidia-*'. Then reboot. - Confirm the operator driver pod is the thing that loads the module.
lsmod | grep nvidiashould appear only after that pod is Running. - If the host driver must stay, I do the opposite in the ClusterPolicy:
driver.enabled: false. The operator then deploys the toolkit and the device plugin, and it stops trying to load a second module.
I do not leave both enabled to "see which one wins". The mismatch comes back on the next reboot. The clean install path is in the operator note.
Commercial Engineering & Platform Infrastructure Advisory
Need hands-on senior engineering to audit, scale, or stabilize your infrastructure?
From bare-metal Kubernetes and immutable Talos Linux to drastically cutting cloud infrastructure bills, resolving PostgreSQL deadlocks, and architecting sovereign AI systems, I work directly with production engineering teams to deliver resilient foundations.