Repository navigation
Serving architecture for tabctx: placement, process model, fit/predict disaggregation (decide by measurement) #39
Description
Why
LLM serving runs one model per node, always on: the weights (60–250 GB) dominate the node. tabctx is different. Its weights are small (TabICLv2 a few hundred MB, TabPFN-3.5 835 MB), and GPU memory goes to long-lived per-dataset contexts of very uneven size, plus fits' transient peaks. Requests are bimodal: fit is heavy and one-off; predict is light, repeated and latency-sensitive. Prod gives tabctx a whole A40 node (4×48 GB); the expected workload is a few concurrent users with small and large tables.
Options
| Design | Strengths | Risks | |
|---|---|---|---|
| A | 1 replica, N GPUs, one engine per GPU with memory- and activity-aware placement (#30, PR #38) | One process, no routing header (keeps the in-place rollout), global placement, shared host RAM | GIL contention across GPU threads (CPU-side preprocessing); one crash loses every GPU's contexts |
| B | N replicas × 1 GPU, Ray Serve session affinity | Process isolation, no GIL, multi-node | Hash ignores memory; needs the header env on the head's proxy (breaks in-place rollout); host RAM split |
| C | B + a memory-aware custom Serve request router | B's isolation with A's placement | Ownership state in the router must survive restarts; complexity |
| D | Disaggregated fit/predict (LLM prefill/decode disaggregation analogue): fit GPUs + predict GPUs, context copied after fit | No head-of-line blocking of predicts by large fits; fit GPUs get full headroom; dense predict packing | PCIe copy of each context (GBs at ~10–25 GB/s); two capacities to tune |
| E | Size tiers inside A: reserve GPU(s) for large tables | Small policy change; protects large-table headroom | Reserved GPUs idle without large tables |
| F | Borrow idle LLM autoscaling headroom | Free capacity | Preemption churn; cluster-level coordination |
Experiments that decide (GKE multi-GPU node, A100-40GB ~ A40)
- GIL scaling (A vs C): fit and predict throughput on 1 vs 4 GPUs in one process (
bench_workload.py,TABCTX_DEVICES=cuda:0vs auto). Near-linear keeps A; flat calls for process-per-GPU. - Head-of-line blocking (D/E): small users' p95 predict latency while large fits run on the same vs a different GPU.
- Context copy cost (D, rebalancing): GPU→GPU and GPU→host bandwidth for real context sizes on the node; the time to move a context compared with re-fitting it.
- Decision write-up with numbers; implement the winner as a follow-up.
Activity
bench: profiling harness (#31), copy bandwidth (#39), GKE GPU devbox
amrit110 commentedon Oct 17, 2026
Platform constraint (2026-10-17): stay serving-framework-agnostic, Dynamo is next
LLM and tabular nodes stay separate for now, but the platform should eventually optimize GPU allocation across both, whatever the backend. inference-platform runs Ray Serve today and is moving to NVIDIA Dynamo (its track: Dynamo 1.3.0, one DGD per model, a shared OpenAI Frontend, fixed replicas; Planner/autoscaling later). What follows for tabctx's design:
serve/factory.pybuilds the engine without Ray). Move the request logic now inserve/app.py(tenant scoping, CSV parsing, uploads, error mapping) into a transport-agnostic service layer, so Ray Serve and a Dynamo worker are both thin adapters.This doesn't change the experiments above. It means that even if A wins on the GIL numbers, the per-device engine must stay separable.
amrit110 commentedon Oct 17, 2026
4×L4 rehearsal (g2-standard-48, prod RayService config, tabctx d56e9e5)
The rollout was in place. Probe 24/24, no GPU OOMs (the device-baseline fix worked), and the replica used all 4 GPUs.
Same 6-user mixed workload (60% 2k×20, 30% 10k×50, 10% 40k×100; 300 s), one replica, 4 GPUs vs pinned to 1:
One process driving 4 GPUs is slower than 1 GPU. Concurrent fits on different GPUs slow each other down by about 2×. Leading suspects:
Next I'll measure this without Ray (
benchmarks/bench_scaling.py: threads vs capped threads vs a process per GPU, on a 4×L4 devbox). That decides the shape of the per-node engine.Two more findings:
Max admitted rows per width (4×L4 sweep, int8 KV):
TabPFN is limited by an uncalibrated estimator. Its calibration (#37) is running on an A100 now.
On an A100 devbox with disk offload, TabICL now fits 130k×500 (65M cells) in 1570 s with 13 GiB resident. 160k×500 is rejected up front with a 413, as designed.
fix(tabicl): one active-context cache per thread on the shared backbone
amrit110 commentedon Oct 17, 2026
Measured, then designed: one worker process per GPU, plus light predicts beside fits
Why concurrent fits on separate GPUs slowed each other down
On 4×A100 (
benchmarks/bench_scaling.py, no Ray), each worker fits the same shape on its own GPU:TabICL fits are host-bound: the driving thread's CPU time is about 90% of wall time (10.2 of 11.3 s at 40000×100). So threads contend for the GIL. Capping intra-op threads changed nothing.
Head-of-line blocking
On one A100 (
benchmarks/bench_overlap.py), predicting on a 2000×20 context:What landed (PR #40 branch)
serve/workers.py(ProcessShard). One worker process per GPU owns that device's engine. Calls are multiplexed over a pipe. Responses carry versioned residency/stats/headroom snapshots, so the router's reads stay lock-free./readyzand Ray'scheck_health, so the replica restarts.TABCTX_WORKER_PROCESSES=1is the default when there is more than one device. Nothing depends on Ray, so a Dynamo worker can host the same core.TabctxEngine:Real fits at the A40/A100 edges succeeded with about 7 GiB peak. A larger calibration grid is being measured now.
Real Ray Serve, 4×A100, 6 users, 300 s (60% 2k×20 / 30% 10k×50 / 10% 40k×100)
Next: review of these commits, an L4 rehearsal of the prod RayService with worker processes, the extended TabPFN grid, and then single-GPU host-overhead work. Fits are host-bound, so CUDA graphs and fewer syncs (#36) are the next big lever.
feat(multidevice): pluggable placement policy and residency events (#39)
feat: multi-GPU replicas with workload-aware placement (0.12.0) (#38)
fix(tabicl): one active-context cache per thread on the shared backbone
feat(multidevice): pluggable placement policy and residency events (#39)
perf: worker processes, compiled TabICL, GPU preprocessing, int8 KV, …
amrit110 commentedon Oct 18, 2026
Close after verifying on the prod A40 (once inference-platform#168 is deployed):
bench_workload.py): fit/predict p50/p95, errors, predicts/sbenchmarks/bench_copy.py) on the A40 node: GPU→GPU and GPU↔host bandwidth, against re-fit time. Still unreported, and it decides option D/v1/tabctx/residencymatches where contexts actually live after the workload