← Back to Blog

A GPU job lands on a CPU node, or a CPU job lands on the GPU node


A GPU node and a plain CPU node in the same rack

Someone says the GPU job is on the wrong machine. I ask which wrong. A pod with no nvidia.com/gpu limit is a CPU pod. The scheduler may place it on the expensive node because that node has CPU free. A pod that does request a GPU and still sits on a CPU node is a different bug: the GPU node is not advertising the resource.

The problem

Without a taint, GPU nodes are normal nodes that happen to have a card. Batch jobs, log collectors, and build pods will land there and steal CPU from inference. Without a resource request, a "GPU job" is not a GPU job. The name in the label does not schedule anything.

The split I enforce

  1. Taint GPU nodes: nvidia.com/gpu=present:NoSchedule.
  2. Every pod that needs the card requests nvidia.com/gpu and tolerates that taint.
  3. Everything else has no toleration, so it stays on CPU nodes.
  4. If a pod requests a GPU and stays Pending, I check allocatable. A taint does not create a device. That case is the zero-GPU note, not this one.

I do not add a nodeSelector on the hostname. Hostnames change. The taint and the resource name do not. Setup of the operator itself is the first note.

Commercial Engineering & Platform Infrastructure Advisory

Need hands-on senior engineering to audit, scale, or stabilize your infrastructure?

From bare-metal Kubernetes and immutable Talos Linux to drastically cutting cloud infrastructure bills, resolving PostgreSQL deadlocks, and architecting sovereign AI systems, I work directly with production engineering teams to deliver resilient foundations.