Fits that run past Ray Serve's 900 s timeout (408): time-aware admission, stop cancelled work #50

On prod, two tabicl-v2 fits were admitted (memory estimate OK) but ran past Ray Serve's request_timeout_s: 900 (inference-platform flux/app/prod/ray-llm-app.yaml:65). Replica log:

• POST /tabctx/v1/tabctx/fit CANCELLED 899995.9ms (500×65,000)
• ... CANCELLED 899997.7ms (1,000×30,000)

The client got a 408 after 15 minutes. Gateway users get the same: its upstream timeouts (920 s) are deliberately longer than Ray's.

Problems

1. Admission checks memory, not time. It admits shapes it can't finish inside the request timeout.
2. Cancellation may not stop the work. Ray cancels the request, but the fit runs in a worker process (perf: worker processes, compiled TabICL, GPU preprocessing, int8 KV, requirement + host admission (0.12.0) #40). If it keeps going, it holds a GPU, a large-fit slot and its host-memory reservation (162.5 GB was held during one of these) for nobody. /readyz hung over the same period (/readyz blocks behind long fits (cancelled after 10 s) #51).
3. No way to run a long fit. No async fit, no keepalive (tabctx#46, item 7).

Plan

• Measure first: after a cancelled fit, does the GPU stay busy, and when are the slot and reservation released?
• Propagate cancellation to the worker (a cancel flag checked between stages or ensemble members, or kill and respawn the worker as the last resort) and release the reservations.
• Time-aware admission. Fit a time model from measured fits, like the memory estimator (rows × features → seconds, per backend and device), and refuse with a clear 413 or 503 code when the estimate exceeds the timeout minus a margin. The message should suggest the async path once it exists.
• Async fit (submit, then poll with retry_in_secs) for fits that can't run synchronously. It can share the dataset-id model: a fit is pending until done, and predict on a pending id returns 409 with Retry-After.

Accept

• No admitted fit ends in 408 in the sweep.
• After a client gives up, GPU, slot and host reservation are released within seconds.
• The time estimate is within 2× on the sweep's shapes.