fix(api): stop an in-flight resume from undoing a paused sandbox delete

[AdaAibaby]

Problem

If a client DELETEs a paused sandbox while a resume for the same sandbox is still in flight, the delete returns 204 and soft-deletes the snapshot, but the resume then publishes the sandbox back as running. The client is told the sandbox is gone while it keeps running until its timeout (and is billable), and the snapshot it would have resumed from has been deleted, so a later pause/resume of that sandbox ends up unrecoverable.

Reproduced deterministically on a dev cluster: create + pause, fire an async resume, then DELETE ~20-60 ms later. DELETE returns 204, resume returns 201, and GET then reports state=running.

Fixes #3636.

Root cause

A paused sandbox has no running-store record — only a snapshot row. So on DELETE:

• StartRemoving reads the running store, gets redis.Nil, and returns ErrNotFound. There is nothing to lock and no execution to pin, so the kill handler records no intent and falls straight through to deleting the snapshot.
• Resume publishes via storage.Add, which is a lockless SET+SADD with no delete-intent check.

The two paths share no lock, so they interleave: the delete can remove the snapshot and return 204 in the same window the resume is restoring the node and about to Add.

This is the same class of stale-write race that RemoveOpts.ExpectExecutionID already fences for running sandboxes (startTransitionScript refuses to overwrite a newer incarnation). The paused case is unprotected precisely because there is no record and no execution ID to pin against.

Fix

Rather than introduce a separate tombstone key with its own TTL and GC, this reuses the reservation a resume already holds for its entire lifecycle (Reserve .. finishStart) as the rendezvous point.

Before touching the snapshot, the kill handler calls ClaimKill, a Lua script that checks the pending set and the storage index atomically:

• A resume is pending, or already finished (back in the index) -> refuse. The handler returns 409 and leaves the snapshot intact; the client retries the kill against the now-running sandbox through the normal locked path.
• Otherwise -> write a short-lived kill-claim and proceed with the snapshot delete. reserveScript now rejects any reservation while that claim exists, so a resume that only starts after we commit to the delete loses too, with a clean 404.

The claim only has to outlive the snapshot soft-delete becoming durable (after that, a resume fails when it fetches the snapshot), so its TTL is short, and it is released eagerly if the delete fails. Net effect: an accepted kill is irreversible — no concurrent resume can publish after it.

Why this shape

• Reuses existing machinery. The claim lives in the reservation namespace, so it inherits the same per-team Redis slot (cluster-atomic with reserveScript) and the same stale-entry lifecycle. No new key family, no new GC path.
• Mirrors the existing CAS intent. It gives the paused-delete path the equivalent of the ExpectExecutionID pin: the write that could resurrect a removed sandbox now checks, atomically, that no kill was accepted first.
• Consistent API behaviour. Returning 409 when a resume is in flight matches what resume itself already returns for a snapshotting sandbox — a transient conflict the client retries.

Testing

• packages/api/internal/sandbox/reservations/redis/kill_claim_test.go — new tests against real Redis (testcontainers): claim succeeds and blocks a later resume; claim refused while a resume is pending; claim refused when the sandbox is already running; release unblocks; and a 50-iteration concurrent Reserve vs ClaimKill race (with -race) asserting the two never both win.
• go build ./..., go vet, and golangci-lint (v2, pinned) all clean.
• Original bug reproduced on the unfixed cluster. A live end-to-end check of the fix on a deployed cluster is still pending; a verification script that asserts the fixed behaviour (409, or 204 and genuinely gone, never 204+running) is ready to run post-deploy.