GPU memory stays taken after the pod is deleted

I delete the notebook pod. A minute later the next job is OOM before it loads a model. On the node, nvidia-smi still lists a process. Kubernetes believes the pod is gone. The card does not.
The problem
Deleting a pod sends SIGTERM, then SIGKILL after terminationGracePeriodSeconds. A Jupyter kernel, a vLLM worker, or a shell someone left in the container can ignore the first signal and outlive the grace period if the grace period is huge, or it can be reparented in a way the device plugin does not reset. The operator does not scrub VRAM on its own.
What I do
- On the node,
nvidia-smiand note the PID. If the PID is not in any container, it is a leftover on the host. - Set a short grace period on GPU pods, 30 seconds is enough for a model server that exits on SIGTERM.
- Add a
preStopthat stops the server. For vLLM that is the process in the container, not a second cleanup script that sleeps. - If a leftover PID remains after the pod is gone, I kill that PID and only then reschedule. I do not "restart the operator" as the first move.
Notebook images are the usual source. I do not run interactive notebooks on the same card as production inference. The setup note is the operator article.
Commercial Engineering & Platform Infrastructure Advisory
Need hands-on senior engineering to audit, scale, or stabilize your infrastructure?
From bare-metal Kubernetes and immutable Talos Linux to drastically cutting cloud infrastructure bills, resolving PostgreSQL deadlocks, and architecting sovereign AI systems, I work directly with production engineering teams to deliver resilient foundations.