Running AI/ML workloads on Kubernetes requires exposing hardware accelerators (GPUs) to pods. Historically, this meant manual driver installations on host nodes. The NVIDIA GPU Operator automates the management of all NVIDIA software components needed to provision GPUs in a Kubernetes cluster.

Prerequisites: Node Preparation

The beauty of the GPU Operator is that your worker nodes need almost nothing installed beforehand. You only need a supported OS (e.g., Ubuntu 22.04), a container runtime (containerd), and a clean state without existing NVIDIA drivers.

# Purge any existing NVIDIA packages to avoid conflicts
sudo apt-get purge -y nvidia* libnvidia*
sudo apt-get autoremove -y

# Verify Nouveau (open-source driver) is loaded temporarily
lsmod | grep nouveau

Annotation: The GPU operator deploys the proprietary NVIDIA driver as a containerized DaemonSet. Any existing host-level drivers will clash, leading to failed pod initializations.

Deploying NFD (Node Feature Discovery)

The operator relies on Node Feature Discovery to identify which nodes actually have GPUs installed, applying labels accordingly so the operator only targets those nodes.

# Add the NVIDIA Helm repository
helm repo add nfd https://kubernetes-sigs.github.io/node-feature-discovery/charts
helm repo update

# Install NFD
helm upgrade --install nfd nfd/node-feature-discovery \
  --namespace node-feature-discovery \
  --create-namespace \
  --set worker.tolerations[0].key="node-role.kubernetes.io/master" \
  --set worker.tolerations[0].operator="Exists" \
  --set worker.tolerations[0].effect="NoSchedule"

Annotation: We use Helm to deploy NFD. The tolerations ensure the NFD worker daemonset runs on all nodes, even control planes, to thoroughly map cluster hardware topologies. Once running, check node labels for feature.node.kubernetes.io/pci-10de.present=true (10de is NVIDIA's vendor ID).

Installing the GPU Operator

With hardware labeled, we deploy the operator. It will orchestrate the Driver, Container Toolkit, Device Plugin, DCGM Exporter, and MIG Manager.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm upgrade --install gpu-operator nvidia/gpu-operator \
  -n gpu-operator --create-namespace \
  --set driver.enabled=true \
  --set toolkit.enabled=true \
  --set devicePlugin.enabled=true \
  --set dcgmExporter.enabled=true \
  --set toolkit.env[0].name=CONTAINERD_CONFIG \
  --set toolkit.env[0].value=/etc/containerd/config.toml \
  --set toolkit.env[1].name=CONTAINERD_SOCKET \
  --set toolkit.env[1].value=/run/containerd/containerd.sock

Annotation: The critical parts here are the toolkit.env variables. Since we are using containerd, the operator needs to know where the configuration file and socket are to dynamically inject the nvidia runtime into the containerd configuration and restart it.

Validating the Deployment

To confirm the cluster can schedule GPU workloads, we run a simple CUDA vector addition pod.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vector-add
spec:
  restartPolicy: OnFailure
  containers:
    - name: cuda-vector-add
      image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubuntu20.04"
      resources:
        limits:
          nvidia.com/gpu: 1
EOF

Annotation: The scheduler sees nvidia.com/gpu: 1 and relies on the NVIDIA Device Plugin to find a capable node. If successful, kubectl logs cuda-vector-add will output Test PASSED, proving the entire driver and runtime injection pipeline succeeded.