fix(deployer): terminate workloads gracefully

[thxCode]

Deleting a workload was an immediate SIGKILL on Docker and Podman, and an unconfigurable 30 seconds
on Kubernetes, so an inference engine never got a chance to drain in-flight requests. This is the gap
the review of gpustack/gpustack#6061
pointed at — it lives in this layer, not in the control plane.

Problem

delete_workload() had no notion of a termination grace period:

• docker.py / podman.py removed every container with remove(force=True), i.e. docker rm -f —
  immediate SIGKILL, no SIGTERM, no grace;
• kuberentes.py never set terminationGracePeriodSeconds on the Pod spec and never forwarded
  gracePeriodSeconds on delete, so the workload silently inherited the API server default with no
  way to configure it.

Reproduced with a container that traps SIGTERM and appends a marker to a bind-mounted file:

[poc] delete took 0.16s (grace=None)
[poc] marker after delete: 'START\n'
[poc] SIGTERM observed: False
[poc] handler completed: False

Change

WorkloadPlan.termination_grace_period_seconds (default 30, the Kubernetes API server default)
is the declarative source of truth, and delete_workload(..., grace_period_seconds=N) is a one-shot
override — mirroring Kubernetes' own terminationGracePeriodSeconds / gracePeriodSeconds pair.

• Docker / Podman stamp the value onto the containers as a label, because neither SDK can express
  a create-time stop timeout at the models layer (stop_timeout is in neither RUN_CREATE_KWARGS nor
  RUN_HOST_CONFIG_KWARGS). Deletion then removes the autoheal sidecar first — it restarts containers
  whose health turns unhealthy, so it would resurrect the very container being drained — stops every
  drainable container concurrently so each gets the whole grace period, removes them, and removes
  the pause container last, since it holds the namespaces the others share.
• Podman passes ignore=True: without it, stop() on an already-exited container answers
  304 Not Modified and podman-py calls response.json() on an empty body, raising
  JSONDecodeError — not an APIError, so it would escape the deletion. INIT containers are always
  already exited, so that path is guaranteed to be hit.
• Kubernetes declares the value on the Pod spec and forwards the override on both the collection
  and the 405 per-Pod fallback paths.
• CLI gains --grace-period-seconds on delete and delete-all.
• A container failing to stop is left to the forceful removal rather than aborting the deletion and
  stranding the pause container and the volumes, and deleting a workload now always raises
  OperationError on all three deployers — neither SDK's errors nor the transport errors underneath
  them share a base class narrow enough to catch on its own (json.JSONDecodeError is not even an
  OSError).

The grace period is not a budget shared between the containers: every one of them is signalled at
once and gets all of it, which is what Kubernetes does. The drain is bounded to 16 workers; a workload
carries a pause container, an autoheal sidecar and a handful of init/run containers, so that ceiling
is not reachable in practice.

Kubernetes stopped recreating unchanged Pods

Comparing the new field in equal_pods surfaced a pre-existing defect: equal_pods never returned
True, so every create_workload() on an existing Pod deleted and recreated it — a model reload on
each deployment. Three normalisations, each measured against a live cluster:

Declared	Stored by the API server	Compared as
hostNetwork: false	omitted	None != False → changed
resources unset	{claims: null, limits: null, requests: null}	an all-null object is truthy, so or {} did not catch it → changed
cpu: 0.5, memory: 1.5Gi	500m, 1536Mi	rewritten spelling → changed

Quantities are now compared by the number they denote rather than by spelling, so any equivalent
spelling reads as unchanged, while a quantity Kubernetes would not accept — underscore separated,
full-width, or whitespace padded, all of which Python's Decimal would happily parse — is refused
rather than passing as unchanged and silently skipping the deployment.

Known limit, left for a separate change: equal_pods still compares the desired spec against the live
object field by field, so it survives only because _create_pod hand-mirrors every API server default
(success_threshold=1 exists purely for that). The shape that removes the whole class is comparing
against the last applied spec recorded on the Pod, the way kubectl apply does.

Test plan

• Unit tests — tests/gpustack_runtime/deployer/test_termination_grace_period.py, 41 cases: call
  order and arguments on both Docker and Podman, the resolution chain (argument → container label →
  30), grace_period_seconds=0 degrading to an immediate kill, a container failing to stop, the
  Kubernetes Pod spec / delete forwarding / transport-failure wrapping, and equal_pods reading a
  rewritten quantity and an omitted toggle as unchanged. Full suite: 536 passed, 20 skipped;
  pre-commit clean.
• Both regression guards were verified by mutating the implementation and re-running: a serial
  drain, and a drain that signals the containers one at a time, each fail the threading.Barrier
  concurrency test while the other cases survive — so the barrier is the load-bearing guard.
• Docker, live on Docker Desktop 29.7.2 — single container: delete took 3.18s,
  SIGTERM observed: True, handler completed: True (0.16s / False / False before). Two RUN
  containers each needing 4s to drain with grace=5: both drain completely, 5.14s.
  grace_period_seconds=0 still kills immediately, 0.19s. End-to-end via the CLI as well.
• Kubernetes, live on a docker-desktop cluster (v1.36.1) —
  spec.terminationGracePeriodSeconds reports the declared value;
  metadata.deletionGracePeriodSeconds reports 30 without an override and 5 with
  grace_period_seconds=5; the container observes SIGTERM while draining. Re-applying an unchanged
  plan no longer recreates the Pod, while changing the grace period or an image still does.
• Podman — not verified against a live daemon, no environment on this machine. Covered by unit
  tests mirroring the Docker ones, plus a targeted check driving podman-py's real Container.stop
  against a synthetic 304 response, which confirms ignore=True is required.

[gemini-code-assist]

Code Review

This pull request introduces support for graceful termination of workloads across Docker, Podman, and Kubernetes deployers. It adds a grace_period_seconds parameter (defaulting to 30 seconds) to the workload deletion commands and APIs. For Docker and Podman, containers are now stopped concurrently using a thread pool to ensure each container receives the full grace period. For Kubernetes, the termination grace period is declared on the Pod spec, and resource quantity parsing is improved to prevent unnecessary pod recreations. Additionally, comprehensive unit tests have been added to verify the new graceful termination behavior. I have no feedback to provide as there are no review comments.

[Copilot]

Pull request overview

This PR adds a configurable termination grace period to workload deletion across Docker, Podman, and Kubernetes deployers so inference workloads can drain in-flight requests before being forcefully torn down, and it fixes Kubernetes pod equality checks to avoid unnecessary pod recreation when the API server normalizes defaults/quantities.

Changes:

• Introduces WorkloadPlan.termination_grace_period_seconds (default 30) plus a delete(..., grace_period_seconds=...) override propagated through the deployer facade, Python API, and CLI.
• Implements graceful termination for Docker/Podman by stopping drainable containers concurrently (bounded concurrency) before force-removing them, and ensures transport/SDK failures are consistently wrapped as OperationError.
• Updates Kubernetes pod creation/deletion to declare/forward grace periods and improves equal_pods/equal_containers to tolerate API-server defaulting and resource quantity rewrites.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/gpustack_runtime/deployer/test_termination_grace_period.py	Adds unit coverage for grace-period resolution/propagation, Docker/Podman drain ordering + concurrency, Kubernetes grace handling, and pod/container equality normalization.
gpustack_runtime/deployer/types.py	Adds the default grace-period constant, plan validation/defaulting, and forwards the delete override through the deployer base API.
gpustack_runtime/deployer/docker.py	Stamps grace period into labels and implements concurrent stop-then-remove deletion with consistent error wrapping.
gpustack_runtime/deployer/podman.py	Mirrors Docker behavior, including ignore=True on stop, label-based grace resolution, and consistent error wrapping.
gpustack_runtime/deployer/kuberentes.py	Declares/forwards grace periods on pod create/delete, wraps transport failures, and fixes equality by normalizing defaults and parsing quantities numerically.
gpustack_runtime/deployer/init.py	Exposes grace_period_seconds in the top-level delete_workload API and forwards it to the selected deployer.
gpustack_runtime/cmds/deployer.py	Adds --grace-period-seconds to delete and delete-all and forwards the value to the runtime API.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.