[NorwinYu]

Problem and result

docs/MULTI_GPU.md says a second card is on par with the faster card alone for decode, and that a third,
slower card makes a pair slower. Both were measured on a fast desktop CPU. A reader with two identical
older cards on a slower workstation CPU cannot tell from that page which side of the line they are on,
and on such a machine the answer is not "on par".

After: one bullet under "Which cards and in what order" that names the condition — the CPU pool is what
the window waits on — with the measured numbers and the mechanism behind them.

Related discussion

The full report and raw data are in the benchmark PR: #3929. No issue: this is a docs
note backed by measurements, not a behaviour change.

Approach

One bullet, docs only. No code and no default is touched. It states:

• the measured pair on two RTX 2080 Ti over a dual-socket Xeon E5-2696 v4 at 180 W, engine 0.1.49:
  decode 35.2 -> 51.0 tok/s, an 11.6K prompt 672 -> 908 tok/s;
• the mechanism from STRATA_DECODE_TIMING=1: the window 70.06 -> 37.96 ms, because the CPU pool falls
  9.52 -> 0.79 ms per window;
• that the "on par" number above it was measured on a fast CPU, so the two statements do not contradict.

Validation

Passed: the numbers are taken from the benchmark report that goes with this change, and the
STRATA_DECODE_TIMING=1 output for both configs was read directly (window, CPU pool and GPU-reach values
above).

Failed: none observed.

Not run: no build is affected, so CUDA, HIP, SYCL and Metal are untouched. No second host was
available to check that the sentence reads correctly for someone who has not seen the report.

Limits

• One measurement, one pair of cards, one CPU. The bullet says "pays most when the CPU pool is the
  bottleneck" rather than giving a threshold, because I did not measure where the crossover is.
• The engine-log window breakdown serialises a little and is not the same statistic as the API-level
  decode numbers next to it; the bullet keeps them in separate sentences.

[Niko1221]

Thank you. The text is in 0.2.7 (docs/MULTI_GPU.md). We could not repeat your two-card runs with a slow CPU, so please try 0.2.7 and tell us whether the page matches your measurements.

Release: https://github.com/Niko1221/Strata/releases/tag/v0.2.7

[ibbobud]

@NorwinYu I have dual Titan Xp pascal cards on a Xeon W-2123 4c/8t cpu if that qualifies as a slow cpu. Let me know if you want me to do any testing.

[coryjett]

A two-card run on a slower CPU than the page's E5-2696 v4, if it helps: Dell T630, 2x Xeon E5-2620 v3 (6C/12T each, 2.4 GHz, AVX2, no AVX-512), 2x RTX 3090 on PCIe 3.0 x16, one card per socket, original Flash-Next IQ3_S, numactl --interleave=all. Same box, same prompts, one card against the split (engine 0.1.48; medians of 3 runs):

	Short prompts, thinking off / on	32K read tok/s	Decode after 32K	64K / 96K read tok/s	Decode after 96K	Expert hits in VRAM
one 3090	~93 / ~77-86	~2,250	~68	1,855-1,887 / 1,920-1,937	61-69	91%
two 3090s, layer split auto	140-145 / 115-126	2,055-2,326	104-114	2,165-2,387 / 2,541-2,754	99-105	99.4%

So on this CPU the split pays about +50% on decode, is flat on a 32K read (the hop costs what the extra residency gains), and +15-40% on 64-96K reads. On 0.2.7 (engine 0.1.52) with --trim-stage-weights at the auto split point the two-card figures are 142-151 / 122-124 short, 2,119-2,173 at 32K, 106-109 after it; the draft-protocol standard run gives 1,127 / 1,610 tok/s prompt at 4K / 16K and 114-129 tok/s decode. A 10-task coding-agent bench went from 3.2 to 2.6 min on the short task and kept its 8.0 min median on the long one with the slow tail gone (worst 9.7 vs 15.6 min). Full report with raw data: #4468 (bench/results/2026-10-28-community-t630-2x3090-pcie3/). One caveat for the page: with a card on each socket, --interleave=all beat --preferred=0 on the 32K read on every start (2,254-2,326 vs 2,055-2,175).

[NorwinYu]

@ibbobud Thanks — that would be very welcome. Yes, please do. The open-source spirit is exactly that: the more people measure, the more useful the page becomes. A 4-core CPU is a different point from either of the two boxes behind it today, so whatever you find is genuinely useful — including "no difference", which is arguably the most informative result of all.

[NorwinYu]

@Niko1221 the 0.2.7 re-run is in #4515 as a benchmark PR. Short version: the page still matches and is now conservative on decode — split 55.5 tok/s (was 51.0), one card unchanged at 34.5, the 11.6K read 962 (was 908), and the split/one-card read ratio is still 1.35x. --trim-stage-weights adds +15% expert slots and +3% on the 11.6K read; IQ3_XXS on the same split gives 69.7 tok/s agent decode.

[coryjett]

A follow-up on the same Dell T630 from my comment above, after a CPU swap: 2x Xeon E5-2620 v3 (6C/12T each) replaced by 2x E5-2697A v4 (16C/32T each, 2.6 GHz), same board, same memory (8x 32 GB DDR4-2133, interleaved over both nodes with numactl --interleave=all), same 2x RTX 3090 split with one card per socket, both at 250 W. Engine 0.1.52, original Flash-Next IQ3_S, --kv-grow --trim-stage-weights, NUMA pool on (2 nodes, one slice per node). Expert hit rate in VRAM is 99.1-99.7% in every run, so this is the case where the CPU pool has the least to do.

Old CPUs against new, same config (2 runs each)

	2x E5-2620 v3 (11 workers, default)	2x E5-2697A v4 (30 workers, default)	Change
Short prompt, thinking off	~147-150 tok/s	~157-165	+8%
Short prompt, thinking on	~122-134	~124-138	+2-3%
32,693-token prompt read	2,774 / 2,941 tok/s	3,079 / 3,145	+8%
Time to first token at 32K	11.1-11.8 s	10.4-10.6 s	about -1 s
Decode after the 32K prompt	117.6 / 117.8 tok/s	119.4 / 116.0	no change

With the experts resident, decode does not move with the CPU; the prompt read and short replies gain about 8%. That fits the page's point from the other side: on this box the GPUs are not waiting on the CPU pool for decode, so a faster CPU only shows up where the CPU still does work.

--pool-workers on the new CPUs

The default picked 30 workers (both sockets' physical cores minus the host core, minus one more). A fresh server per value, two speed checks each, same prompts:

Workers	32K read tok/s (runs)	Decode after 32K tok/s (runs)
30 (default)	3,014 / 3,091	117.8 / 119.1
30 (again, at the end of the sweep)	2,965 / 3,205	119.3 / 130.9
22	3,030 / 2,954	129.7 / 119.1
18	3,064 / 3,116	122.0 / 134.8
17	2,859 / 3,154	119.5 / 124.7
15	3,201 / 3,150	126.6 / 130.2
15 (again)	3,154 / 3,132	120.1 / 116.8
11 (the old CPUs' default)	3,188 / 3,167	118.0 / 123.1

Run-to-run noise is about ±4-5% (the two 30-worker sessions alone span 2,965-3,205), so every value from 11 to 30 is within noise of the others. 15 was the most consistent on the 32K read (four runs, 3,132-3,201) and I kept it, but I would not call it a measured win. On a box with nearly every expert in VRAM, the worker count does not matter much; the larger gains in the DETAILS.md table (#3201) were on boxes with more experts on the CPU.

Not run

No STRATA_DECODE_TIMING=1 pair on the new CPUs, so I don't have the CPU-pool share of the window to put next to the page's 9.52 -> 0.79 ms. No one-card arm on the new CPUs.