Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/MULTI_GPU.md
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,14 @@ The Coder on an RTX 5080 + RTX 3090 (Ryzen 9 9950X3D), 32K context; details in
- Leave out a much slower card when two already hold the model. An RTX 2080 Ti as a third card made the 5080 +
3090 pair slower (68 / 90 tok/s decode): every extra card costs its own round per window.
- More cards pay off when the model's routed experts do not fit the faster ones.
- **The split pays most when the CPU pool is the bottleneck.** On two identical RTX 2080 Ti
22 GB over a dual-socket Xeon E5-2696 v4 (AVX2 only, 88 logical CPUs), a plain split moved
decode from 35.2 to 51.0 tok/s (+45%) and an 11.6K-token prompt from 672 to 908 tok/s (+35%)
at 180 W, engine 0.1.49, six 300-token coding prompts. `STRATA_DECODE_TIMING=1` shows why:
the window fell from 70.06 to 37.96 ms, almost entirely because the CPU pool fell from 9.52
to 0.79 ms per window. The "on par" number above is a fast CPU (9950X3D); when the CPU pool
is what the window waits on, moving half the layers to a second identical card removes that
work and pays about 1.45x.

## One conversation with both cards busy (`--pipeline-windows`, default on two or more AMD gfx12 stages)

Expand Down
Loading