Provisioning the NVIDIA GPU Operator for K8s AI Workloads

Running AI and machine learning workloads on Kubernetes requires exposing host GPUs to pods. Manually installing drivers across every node is inefficient and error-prone. The NVIDIA GPU Operator automates this process by managing drivers, container runtimes, and monitoring tools directly within the cluster as containerized components.

Prerequisites: Node Preparation

Worker nodes require minimal host setup. Use a supported Linux distribution (such as Ubuntu 22.04) with containerd configured as the container runtime, and ensure no existing NVIDIA drivers are installed on the host.

# Purge any existing NVIDIA packages to avoid conflicts
sudo apt-get purge -y nvidia libnvidia
sudo apt-get autoremove -y
# Verify Nouveau (open-source driver) is loaded temporarily
lsmod | grep nouveau

Note: The GPU Operator deploys the proprietary NVIDIA driver as a containerized DaemonSet. Any existing host-level NVIDIA drivers will conflict with these containers and cause pod initialization failures.

Deploying Node Feature Discovery (NFD)

The GPU Operator relies on Node Feature Discovery (NFD) to identify GPU-equipped nodes and label them automatically, ensuring workloads and operator components target the appropriate hardware.

# Add the NVIDIA Helm repository
helm repo add nfd https://kubernetes-sigs.github.io/node-feature-discovery/charts
helm repo update
# Install NFD
helm upgrade --install nfd nfd/node-feature-discovery \
 --namespace node-feature-discovery \
 --create-namespace \
 --set worker.tolerations[0].key="node-role.kubernetes.io/master" \
 --set worker.tolerations[0].operator="Exists" \
 --set worker.tolerations[0].effect="NoSchedule"

Note: Tolerations allow NFD workers to run across all nodes, including control plane nodes, to detect hardware capabilities. Once deployed, verify that GPU nodes receive the label feature.node.kubernetes.io/pci-10de.present=true (10de is NVIDIA's PCI vendor ID).

Installing the GPU Operator

After labeling GPU nodes, install the GPU Operator using Helm. It orchestrates the Driver, Container Toolkit, Device Plugin, DCGM Exporter, and MIG Manager.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm upgrade --install gpu-operator nvidia/gpu-operator \
 -n gpu-operator --create-namespace \
 --set driver.enabled=true \
 --set toolkit.enabled=true \
 --set devicePlugin.enabled=true \
 --set dcgmExporter.enabled=true \
 --set toolkit.env[0].name=CONTAINERD_CONFIG \
 --set toolkit.env[0].value=/etc/containerd/config.toml \
 --set toolkit.env[1].name=CONTAINERD_SOCKET \
 --set toolkit.env[1].value=/run/containerd/containerd.sock

Note: The toolkit.env parameters instruct the operator where to locate containerd's configuration file and socket. This enables it to inject the nvidia-container-runtime into containerd and restart the service automatically.

Validating the Deployment

Verify GPU scheduling by deploying a sample CUDA vector addition pod.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
 name: cuda-vector-add
spec:
 restartPolicy: OnFailure
 containers:
 - name: cuda-vector-add
 image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubuntu20.04"
 resources:
 limits:
 nvidia.com/gpu: 1
EOF

Note: When requested (nvidia.com/gpu: 1), the Kubernetes scheduler assigns the pod to a GPU node using the NVIDIA Device Plugin. Operate kubectl logs cuda-vector-add to verify the output receiving Test PASSED confirms proper driver and runtime configuration.