← Back to Blog

GPU memory stays taken after the pod is deleted


GPU server chassis after a job, card still seated

I delete the notebook pod. A minute later the next job is OOM before it loads a model. On the node, nvidia-smi still lists a process. Kubernetes believes the pod is gone. The card does not.

The problem

Deleting a pod sends SIGTERM, then SIGKILL after terminationGracePeriodSeconds. A Jupyter kernel, a vLLM worker, or a shell someone left in the container can ignore the first signal and outlive the grace period if the grace period is huge, or it can be reparented in a way the device plugin does not reset. The operator does not scrub VRAM on its own.

What I do

  1. On the node, nvidia-smi and note the PID. If the PID is not in any container, it is a leftover on the host.
  2. Set a short grace period on GPU pods, 30 seconds is enough for a model server that exits on SIGTERM.
  3. Add a preStop that stops the server. For vLLM that is the process in the container, not a second cleanup script that sleeps.
  4. If a leftover PID remains after the pod is gone, I kill that PID and only then reschedule. I do not "restart the operator" as the first move.

Notebook images are the usual source. I do not run interactive notebooks on the same card as production inference. The setup note is the operator article.

Commercial Engineering & Platform Infrastructure Advisory

Need hands-on senior engineering to audit, scale, or stabilize your infrastructure?

From bare-metal Kubernetes and immutable Talos Linux to drastically cutting cloud infrastructure bills, resolving PostgreSQL deadlocks, and architecting sovereign AI systems, I work directly with production engineering teams to deliver resilient foundations.