Repository navigation
Run one large fit's ensemble members across idle GPUs #48
Copy link
Copy link
Open
Labels
P1High: next upHigh: next uparea/latencyFit and predict timeFit and predict timearea/multi-gpuSeveral GPUs per replica: placement, workers, parallel fitsSeveral GPUs per replica: placement, workers, parallel fitsenhancementNew feature or requestNew feature or request
Description
A fit runs on one GPU even when the other three are idle. On prod (4× A40), tabicl-v2 fits take 112 s (500×30,000), 138 s (500×45,000) and 130 s (10×100,000), and two shapes ran past Ray Serve's 900 s request timeout (#50).
TabICL and TabPFN both ensemble n_estimators members (8 by default). The members are independent until their outputs are averaged. Splitting them across free GPUs (2 per GPU on 4 GPUs) should cut large-fit time by about 3–4×, without changing the result (same members, same seeds, averaged the same way).
Design questions
- Context layout. Each member's KV (or representation) cache stays on the GPU that built it, and predict fans out to the same GPUs and averages. Alternatively, gather onto one GPU after the fit. That trades predict latency against cache placement and memory accounting (placement.py, cache manager).
- When to split. Only when the estimated fit time is large (say over 10 s) and the GPUs have free capacity and no queued work. A small fit must never wait for, or be split across, busy GPUs. Placement events (feat: multi-GPU replicas with workload-aware placement (0.12.0) #38) already have the information.
- Admission and memory. The per-GPU peak drops (fewer members per GPU), so larger tables can fit. The estimator needs per-member terms.
- Interaction with concurrency (Concurrent large fits are slower on 4 GPUs than on 1 (host contention?) #41) and host memory (offload staging per GPU).
- Worker processes (perf: worker processes, compiled TabICL, GPU preprocessing, int8 KV, requirement + host admission (0.12.0) #40): one fit spanning several workers.
Accept
- The same predictions as the single-GPU fit (bit-identical, or within tolerance with the reason documented).
- Large fit time ≥2.5× better on 4 GPUs for 500×30,000 and 10×100,000.
- Small and medium fits unchanged (p50 and p95) under the mixed workload (
bench_workload.py).
Relates to #35 (context parallelism for tables bigger than one GPU), #39.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
P1High: next upHigh: next uparea/latencyFit and predict timeFit and predict timearea/multi-gpuSeveral GPUs per replica: placement, workers, parallel fitsSeveral GPUs per replica: placement, workers, parallel fitsenhancementNew feature or requestNew feature or request
Activity
[-]P1 latency: run one large fit's ensemble members across idle GPUs[/-][+]Run one large fit's ensemble members across idle GPUs[/+]