Skip to content

Run one large fit's ensemble members across idle GPUs #48

Open

Description

@amrit110

A fit runs on one GPU even when the other three are idle. On prod (4× A40), tabicl-v2 fits take 112 s (500×30,000), 138 s (500×45,000) and 130 s (10×100,000), and two shapes ran past Ray Serve's 900 s request timeout (#50).

TabICL and TabPFN both ensemble n_estimators members (8 by default). The members are independent until their outputs are averaged. Splitting them across free GPUs (2 per GPU on 4 GPUs) should cut large-fit time by about 3–4×, without changing the result (same members, same seeds, averaged the same way).

Design questions

Accept

  • The same predictions as the single-GPU fit (bit-identical, or within tolerance with the reason documented).
  • Large fit time ≥2.5× better on 4 GPUs for 500×30,000 and 10×100,000.
  • Small and medium fits unchanged (p50 and p95) under the mixed workload (bench_workload.py).

Relates to #35 (context parallelism for tables bigger than one GPU), #39.

Activity

  1. added
    enhancementNew feature or request
    P1High: next up
    area/multi-gpuSeveral GPUs per replica: placement, workers, parallel fits
    on Oct 20, 2026
  2. changed the title [-]P1 latency: run one large fit's ensemble members across idle GPUs[/-] [+]Run one large fit's ensemble members across idle GPUs[/+] on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High: next uparea/latencyFit and predict timearea/multi-gpuSeveral GPUs per replica: placement, workers, parallel fitsenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions