docs: redraw architecture diagram for 0.12.0 (multi-GPU router, worker processes, host guard)

CHANGELOG.md

@@ -2,7 +2,7 @@
 
 ## [Unreleased]
 
-## [0.12.0] - unreleased
+## [0.12.0] - 2026-10-18
 ### Added
 - **Multi-GPU replicas** (#30). `MultiDeviceEngine` runs one `TabctxEngine` per device -- backends bound to `cuda:i`, its own cache, lock and spill subdirectory -- behind the single-engine API, built automatically when a replica sees several GPUs (`TABCTX_DEVICES=auto`, or an explicit list). Placement is aimed at the expected workload, a few concurrent users with small and large tables: a fit goes where its estimated peak fits without eviction, on the least active device (in-flight calls, then calls in the last 30s), else to the device with the most headroom. Predicts route to the device holding the context (resident first, then spilled), found from lock-free residency/spill snapshots, so routing never waits on another GPU's work and fits and predicts on different devices run concurrently. Per-dataset_id locks serialize fits and evicts of one id (no duplicate copies from retried fits); a re-fit lands on its device before other copies are removed, so a failed re-fit leaves the previous context serving. Unknown models are rejected (422) before anything changes. `TABCTX_SPILL_CAPACITY_BYTES` is the replica's total, split across devices' spill subdirectories; a spill layout change (one vs several devices) is logged. `TABCTX_DEVICES` rejects mixed CPU/GPU lists, a bare `cuda` among indexed GPUs, and indices the process can't see. `/readyz` gains a per-device breakdown (contexts, bytes, activity, real GPU memory); `/v1/tabctx/limits` reports the best single device. A single GPU builds exactly the engine it did before.
 - **One worker process per GPU** (#39, `TABCTX_WORKER_PROCESSES`, default on for multi-GPU replicas). Fits are host-bound, so threads of one process driving several GPUs contend for the GIL: on 4x A100, four concurrent fits ran at 2.1-3.1x aggregate from threads and 3.96-4.3x from processes (`benchmarks/bench_scaling.py`). `serve/workers.py` runs each device's engine in its own process behind a `ProcessShard` proxy; the router reads versioned residency/headroom snapshots that every reply carries, so its decisions stay lock-free. A dead worker fails its calls, is skipped by placement, and fails `/readyz` and Ray Serve's `check_health` so the replica restarts. Nothing depends on Ray. Under Ray Serve on 4x A100 with 6 mixed users: 2.58 -> 5.0 predicts/s, small-fit p50 1.6 -> 0.6 s.

README.md

@@ -90,12 +90,17 @@ deployment is exactly fast enough.
 
 *Figure: a request enters through Ray Serve's consistent-hash proxy
 (affinity on the session header) into a `TabctxService` replica, which
-resolves the tenant, parses inline JSON or an uploaded CSV, and hands the
-table to the engine. The engine validates input, gates admission on an
-estimated memory peak, keeps encoded contexts in a byte-budgeted LRU
-cache, and dispatches to whichever backend fit the context. Evicted
-contexts optionally spill to local disk and reload on their next use
-(editable source: `docs/figures/architecture.svg`).*
+resolves the tenant, parses inline JSON or an uploaded CSV, and coalesces
+same-context predicts. With several GPUs, a `MultiDeviceEngine` places
+each fit on one GPU and routes predicts to the GPU holding the context,
+from lock-free snapshots; each GPU's engine runs in its own worker
+process. A GPU's engine validates input, admits a fit on its estimated
+memory need, keeps encoded contexts in a byte-budgeted LRU cache, and
+dispatches to whichever backend fit the context. Evicted contexts
+optionally spill to local disk and reload on their next use, and one
+host-memory guard admits large fits against the pod's RAM across all
+its workers. A single-GPU replica calls its one engine directly (editable
+source: `docs/figures/architecture.svg`).*
 
 The unit everything revolves around is a **context**: one training
 table, encoded once by one model, addressed by `dataset_id` (scoped per
@@ -418,8 +423,16 @@ pip install "tabctx[dev]"        # + test dependencies
 - **Same-context coalescing** (`CoalescingPredictor`): concurrent
   requests against one cached context are packed into a single GPU call
   within a ~5ms window and split back per caller, amortizing per-call
-  overhead without ever running concurrent GPU work (the memory budget's
-  single-in-flight-call assumption is preserved).
+  overhead without adding concurrent GPU calls (each GPU still runs at
+  most one predict at a time, beside at most one fit).
+- **Several GPUs per replica** (`MultiDeviceEngine`): one engine per GPU,
+  each in its own worker process, behind the single-engine API; see
+  [Several GPUs in one replica](#several-gpus-in-one-replica).
+- **Host-memory admission** (`HostMemoryGuard`): large TabICL fits stage
+  tens of GiB in host RAM, so calls of 1 GiB or more reserve their
+  estimated host bytes against one budget shared by the pod's workers (first
+  come, first served), so two huge fits queue instead of getting the pod
+  OOM-killed with every tenant's context.
 
 ## Validated at scale (real A100-40GB, not simulated)
 
@@ -453,9 +466,11 @@ sharing one A100-40GB** (v0.7.0, 2026-08-29,
 
 ## Status & Roadmap
 
-v0.10.0: multi-model endpoint, calibrated peak-aware admission, disk
-spillover and multi-replica routing have all shipped and been validated
-on real GPUs (see [CHANGELOG.md](CHANGELOG.md)). Open roadmap items are
+v0.12.0: multi-model endpoint, calibrated peak-aware admission, disk
+spillover, multi-replica routing and multi-GPU replicas with a worker
+process per GPU have all shipped and been validated on real GPUs (see
+[CHANGELOG.md](CHANGELOG.md)); 0.12.0 serves on Vector's 4x A40 prod
+node. Open roadmap items are
 tracked as GitHub issues labelled `roadmap`. **Read
 [ROADMAP.md](ROADMAP.md) before starting new work** -- it ranks what's
 next and why.