← Back to Blog

DCGM shows GPU use and not which pod holds the card


Status LEDs on a GPU server bezel, no readable text

The GPU is busy. The dashboard says so. It does not say which pod. I end up SSH-ing to the node and running nvidia-smi, which is the thing the exporter was supposed to replace.

The problem

DCGM knows the device. It does not know Kubernetes until the exporter reads the kubelet pod-resources socket and attaches the pod and namespace as labels. If that socket is not mounted, or the exporter flag is off, you get device metrics with an empty pod label.

The fix

In the ClusterPolicy I keep dcgmExporter.enabled: true. I check the exporter pod has the pod-resources volume. Then I look for a metric that carries pod and namespace:

DCGM_FI_DEV_GPU_UTIL{pod!="",namespace!=""}

If the series exists and pod is always empty, the socket is there and the mapping is not. I check the exporter logs for the kubelet socket path before I build a second dashboard. A new panel on an unlabeled metric still will not name the pod.

Once the label is there, a normal PromQL query answers "who is on this card" without a shell. The operator install is the setup note.

Commercial Engineering & Platform Infrastructure Advisory

Need hands-on senior engineering to audit, scale, or stabilize your infrastructure?

From bare-metal Kubernetes and immutable Talos Linux to drastically cutting cloud infrastructure bills, resolving PostgreSQL deadlocks, and architecting sovereign AI systems, I work directly with production engineering teams to deliver resilient foundations.