diff --git a/experimental/CollectiveX/README.md b/experimental/CollectiveX/README.md index 5be2797487..89539a6a66 100644 --- a/experimental/CollectiveX/README.md +++ b/experimental/CollectiveX/README.md @@ -39,8 +39,11 @@ in one of two modes: `kNumMaxWarpGroups` is 16, since upstream raised `kNumMaxTopK` 9 -> 16 (uccl#1016, 2026-07-13) and our pin is six days later. The product is 16 for every CU count, so this is a dated regression rather than a hardware limit, and the AMD SKUs keep UCCL-EP normal mode without LL), - and NCCL EP EP8 on all six NVIDIA SKUs, restored once the single-handle fix removed the - [NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303) signal aliasing that had wedged them. + and NCCL EP at EP8 on H100/H200/B300 and at EP8 *and EP16* on B200/GB200/GB300, enabled with the + v0.2 upgrade (the `nccl-extensions` wheel ships the combine-recv fence whose absence had held + these rows — see the backend table below; the earlier + [NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303) signal aliasing was already fixed + by the single-handle change). B300 carries the `candidate` NCCL EP as its *only* low-latency row, so it has no production decode coverage. DeepEP V2 emits no LL row at all on B300 (the IBGDA address-handle wall in the backend table below), and `_ll_runnable` adds only runnable cells, so that wall is prose here @@ -131,7 +134,7 @@ scale-up domain. | DeepEP V2 | `production`, with vLLM `--all2all-backend deepep_v2`, SGLang `--moe-a2a-backend deepep` | `normal` mode is PR #605 `ElasticBuffer` plus exact upstream #630 and #640 fixes: LSA for scale-up and GIN for x86 EP16 scale-out. FP8 dispatch via `use_fp8_dispatch` (blockwise e4m3fn) alongside BF16. `low-latency` mode is the legacy `deep_ep.Buffer` IBGDA decode kernels (per-expert padded layout, weighted combine, `use_fp8` e4m3fn), decode only, with EP8 wherever enabled, plus EP16 on GB200/GB300 (inside the MNNVL domain) and on B200's nscale bare-metal pool (IBGDA over native IB rails with `/dev/gdrdrv`, although the prior virtualized b200 pool could never run it). B300 is an unsupported coverage row in `low-latency`: the legacy Buffer self-enables NVSHMEM IBGDA even for a single-node EP8 run, and on B300 address-handle creation fails (`ibgda.cpp:2234 Unable to create ah`), rc255 on all eight ranks. `NVSHMEM_DISABLE_IB=1` does not help. The Buffer re-enables IBGDA regardless, and the run fails identically with it set and unset (measured on b300-002 and b300-011) | | MoRI | `production`, with vLLM `--all2all-backend mori_*`, SGLang `--moe-a2a-backend mori` | `normal` mode uses the direct `IntraNode` kernel for scale-up EP8 on every CDNA SKU. EP16 remains an unsupported coverage row on all three CDNA SKUs. Part of the old ROCm/mori#475 corruption was this harness passing dispatch's returned recv-slot indices to `combine()` instead of the rank's own routing (root-caused upstream, guarded by ROCm/mori#546, kernels unchanged) — with the corrected call, single-shot InterNodeV1 is clean through T=512 on mi355x — but a residual stochastic corruption remains from T~128 up under repeated execution and is near-certain at prefill sizes (run 33045314017/33050026476; unaffected by per-pair drains, so not a buffer-reuse race; upstream cannot reproduce on ionic driver 26.03 vs our 25.11). The tw pairs additionally have no cross-node GPU fabric, and mi355x EP16 has no publishable transport today: uccl-ep's CPU-proxy RDMA is functional on Pollara but ~13x under its documented bandwidth (~6 GB/s vs 82; unchanged by registration mode or traffic class — ionic-driver suspect, same 25.11-vs-26.03 delta as the mori residual). `low-latency` mode selects the `AsyncLL` decode kernel driven split-phase (dispatch send+recv, combine send+recv — the kernel SGLang deploys; same compact layout and unweighted combine as `IntraNode`; earlier rows measured `IntraNodeLL`, discriminated by `kernel_generation`), decode/EP8 only. FP8 dispatch is caller-prequantized (per-SKU e4m3fnuz on gfx942, e4m3fn on gfx950). Combine stays BF16 (`quant_type=none`) alongside BF16 dispatch | | UCCL-EP | `candidate` (no engine exposes a UCCL-EP selector) | [UCCL](https://github.com/uccl-project/uccl) EP: a drop-in, API-identical DeepEP replacement whose CPU proxies issue GPUDirect RDMA over plain `libibverbs` (no NVSHMEM/IBGDA), with software message ordering, atomics, and flow control. Scale-up is single-node `cudaIpc` over NVLink/XGMI (never MNNVL). `normal` mode is the legacy `Buffer` `dispatch`/`combine` (unweighted rank-sum). `low-latency` reuses the legacy `low_latency_dispatch`/`low_latency_combine` decode kernels (weighted combine), decode/EP8 only. FP8 dispatch is caller-prequantized in `normal` mode (blockwise e4m3fn, per-SKU e4m3fnuz on gfx942). In `low-latency` mode the caller sends BF16 and the decode kernel quantizes to e4m3 internally (`use_fp8`). Combine is BF16. Runs on NVIDIA and AMD (H100/H200/B200 + MI300X/MI325X/MI355X), EP8 scale-up. Cross-node EP16 is functional (the internode RDMA path connects and the light case passes correctness) but its CPU-proxy throughput overruns the standardized per-case wall-clock budget on heavy token counts, so EP16 is an unsupported coverage row for now | -| NCCL EP | `candidate` (NVIDIA's own library, but no engine exposes an NCCL-EP selector) | [NCCL EP](https://github.com/NVIDIA/nccl/tree/master/contrib/nccl_ep): NVIDIA's native MoE dispatch/combine on the NCCL Device API, using LSA (NVLink load/store) intra-node and GIN (GPU-Initiated Networking) inter-node, driven through the `nccl4py` bindings. `normal` mode selects the `HIGH_THROUGHPUT` algorithm (FLAT `[N, hidden]` receive, unweighted rank-sum combine). The `LOW_LATENCY` algorithm carries an EP8 `ll_backends` row on all six NVIDIA SKUs, restored once the single-handle fix removed the NVIDIA/nccl#2303 signal aliasing. That LL decode ladder is clamped to T<=128, below its 256-slot receive: `nccl_ep`'s combine recv pipeline is a port of DeepEP's pre-#642 kernel and is missing the same shared-memory fence before `mbarrier_arrive`, which corrupted T=256 on GB300 in 1 of 5 executions. It was bimodal, with healthy rows at 0.0039 relative error against 0.4704 on the failure. The fence is absent at NVIDIA/nccl master, so it is unfixed upstream. The clamp lowers exposure and is **not** a safety boundary: the fence is missing on every combine recv and T=256 is merely the rung with the most pipeline iterations, so lower rungs are less likely to hit the race rather than immune. Restore when a fixed wheel ships BF16 only: `contrib/nccl_ep/RELEASE.md` says "No FP8 support", so no FP8 case is emitted. That note is worth re-testing rather than trusting, because the C library at our pinned commit does read `inputs->scales` and switch on e4m3/e5m2, the two documented FP8 exclusions are expert-major layouts we do not use, and `NVIDIA/nccl` has not moved since 2026-06-11 while `NVIDIA/nccl-extensions` has replaced that row outright. NVIDIA-only and CUDA 13 only. EP8 scale-up on H100/H200/B200/B300 plus EP8 and EP16 on GB200/GB300, where EP16 stays inside the MNNVL scale-up domain. x86 EP16 scale-out is an unsupported coverage row: the cross-node GIN path faults inside `nccl_ep.cc` identically on RoCE and IB across four SKUs, a GDAKI limit rather than a fabric-selection one | +| NCCL EP | `candidate` (NVIDIA's own library, but no engine exposes an NCCL-EP selector) | [NCCL EP v0.2](https://github.com/NVIDIA/nccl-extensions): NVIDIA's native MoE dispatch/combine on the NCCL Device API, using LSA (NVLink load/store) intra-node and GIN (GPU-Initiated Networking) inter-node. Since v0.2 it ships as the `nccl-extensions` PyPI wheel (which owns `nccl.ep`; `nccl4py` — pinned alongside — provides only `nccl.core`), replacing the `contrib/nccl_ep` tree in `NVIDIA/nccl`, which has not moved since 2026-06-11. `normal` mode selects the `HIGH_THROUGHPUT` algorithm (FLAT `[N, hidden]` receive, unweighted rank-sum combine). The `LOW_LATENCY` algorithm carries `ll_backends` rows at EP8 on H100/H200/B300 and EP8+EP16 on B200/GB200/GB300. Its decode ladder runs unclamped to the full 256-slot receive: v0.1's combine recv pipeline was a port of DeepEP's pre-#642 kernel, missing the shared-memory fence before `mbarrier_arrive` (bimodal T=256 corruption on GB300, 1 of 5 executions at 0.4704 relative error against healthy 0.0039), and the v0.2 wheel ships that fence (`fence_view_async_shared` before the `emptyBarriers` arrive in the shipped `ll_ep.cuh`), the documented restore condition for the T<=128 clamp it forced. BF16 only for now: v0.2 exposes FP8 dispatch (`DS_FP8E3M4`) and an experimental NVFP4 combine recipe, but wiring those into the fp8_consume model is separate bring-up, so no FP8 case is emitted yet. NVIDIA-only and CUDA 13 only. EP8 scale-up on H100/H200/B300, EP8 and EP16 on B200 (v0.2 fixes the RDMA EP16 GIN fault this pool hit under v0.1) and on GB200/GB300, where EP16 stays inside the MNNVL scale-up domain. x86 EP16 scale-out elsewhere (H100/H200/B300) remains an unsupported coverage row pending revalidation: under v0.1 the cross-node GIN path faulted inside `nccl_ep.cc` identically on RoCE and IB, and only the B200 configuration is claimed fixed | | FlashInfer EP | `production`, with vLLM `--all2all-backend flashinfer_nvlink_one_sided` | [FlashInfer](https://github.com/flashinfer-ai/flashinfer) `MoeAlltoAll`: TensorRT-LLM's one-sided MNNVL all-to-all, where each rank writes tokens straight into its peers' workspace windows and combine reads them back, with no send/recv pairing and no NVSHMEM. `normal` mode only (there is one kernel family and no separate decode path), and GB200/GB300 only, since the transport is MNNVL. FP8 dispatch is caller-prequantized blockwise e4m3fn, carried as a fourth dispatch payload alongside its per-128-block FP32 scales, with the combine plane forced to BF16. The C++ `toNvDataType` accepts only fp16/bf16/fp32 for combine, so an FP8 combine buffer would raise rather than corrupt. EP8 and EP16, both inside the scale-up domain. Unlike every other backend here, its combine accumulates in the PAYLOAD dtype rather than FP32: wheels before 0.6.16 reduce the top-k contributions with a pairwise BF16 tree that rounds at every level, so the oracle models that reduction directly (`combine_reduction = "topk-slot-tree"`) instead of widening the tolerance. 0.6.16 moved the accumulator to FP32, and the adapter switches models on the installed version | DeepEP V2 means the `ElasticBuffer` implementation introduced by diff --git a/experimental/CollectiveX/bench/ep_nccl.py b/experimental/CollectiveX/bench/ep_nccl.py index be425fdf90..df3f9bc75f 100644 --- a/experimental/CollectiveX/bench/ep_nccl.py +++ b/experimental/CollectiveX/bench/ep_nccl.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 """NCCL EP adapter: NVIDIA's native MoE dispatch/combine on the NCCL Device API. -NCCL EP (github.com/NVIDIA/nccl/contrib/nccl_ep, arXiv 2603.13606) is a ground-up MoE +NCCL EP (github.com/NVIDIA/nccl-extensions, arXiv 2603.13606) is a ground-up MoE communication library built on NCCL's Device API — LSA (NVLink load/store) intra-node and GIN (GPU-Initiated Networking) inter-node — with two algorithms selected per case: normal -> HIGH_THROUGHPUT (HT), the Hybrid-EP-derived prefill/train path. FLAT recv @@ -13,19 +13,21 @@ Because both map exactly onto the two combine contracts CollectiveX already models, the scale_up_domain two-level combine oracle in ep_harness applies unchanged. -BF16 only: `contrib/nccl_ep/RELEASE.md` says "No FP8 support", so this adapter does not -override the FP8 encode hooks (SUPPORTED_PRECISIONS=("bf16",)). Re-test before trusting that -note — the C library at our pinned commit reads `inputs->scales` and switches on e4m3/e5m2, -and the two documented FP8 exclusions are expert-major layouts we do not use. +BF16 only for now: v0.2 grows a real quantization surface (DispatchQuantizationRecipe.FWD / +DS_FP8E3M4 for FP8 dispatch, experimental CombineQuantizationRecipe.NVFP4), but wiring it into +the fp8_consume model is its own bring-up, so this adapter still does not override the FP8 +encode hooks (SUPPORTED_PRECISIONS=("bf16",)). Communicator bootstrap: NCCL EP forms its OWN NCCL communicator (separate from PyTorch's process group) via ``Communicator.init(nranks, rank, unique_id)``. Upstream broadcasts the unique id with MPI; CollectiveX has no MPI, so rank 0 generates the id and we broadcast its bytes over the already-initialized torch process group (see ``_bootstrap_comm``). -Python bindings are ``nccl4py`` (``import nccl.core`` + ``nccl.ep``); the API surface used -here is verified against upstream ``bindings/nccl4py/nccl/ep`` and driven exactly as -``contrib/nccl_ep/ep_test.py`` drives it. +Python bindings are split across two wheels since v0.2: ``nccl-extensions`` owns ``nccl.ep`` +(the libnccl_ep.so JIT runtime + Cython bindings) and ``nccl4py`` provides ``nccl.core`` +(Communicator/UniqueId). The API surface used here is verified against the published +nccl-extensions wheel and driven exactly as upstream's ep_test.py drives it — every class and +signature this adapter touches is unchanged from v0.1. """ from __future__ import annotations @@ -69,23 +71,16 @@ # token counts are measured. Separating them lets the ladder be clamped around a kernel defect # without moving the footprint and silently re-basing the rungs that remain. # -# The ladder sits below the buffer because nccl_ep's low_latency.cu is a port of DeepEP's -# PRE-FIX low-latency combine: in the combine recv pipeline the reduction warps read shared -# memory and then mbarrier_arrive(emptyBarriers[stageIdx]) with no fence.proxy.async.shared::cta -# between, so the producer's next TMA load can overwrite a stage while consumer reads are still -# in flight. DeepEP closed exactly this with a one-line fence in PR #642; the fence is absent -# both at our pin and at NVIDIA/nccl master, so it is unfixed upstream. -# -# Observed on gb300 EP8 BF16 at T=256: 1 failure in 5 executions, bimodal -- healthy rows give -# max relative error 0.0039, the failure gave 0.4704, with nothing between, which is a discrete -# corrupted write rather than tolerance noise. -# -# THIS CLAMP IS NOT A SAFETY BOUNDARY. The fence is missing on every combine recv; T=256 is only -# the rung with the most pipeline iterations, and the receive plane is not even full there. Lower -# rungs are LESS LIKELY to hit the race, not immune. Restore _LL_LADDER_CAP to _LL_BUFFER_CAP -# once a fixed wheel ships. +# The ladder is RESTORED to the full buffer under nccl-ep v0.2. The v0.1 port of DeepEP's +# low-latency combine lacked the PR #642 fence — reduction warps read shared memory, then +# mbarrier_arrive(emptyBarriers[stageIdx]) with no fence between, letting the producer's next +# TMA load overwrite a stage mid-read (observed on gb300 EP8 BF16 at T=256: bimodal 1-in-5 +# discrete corruption, max rel err 0.4704 vs healthy 0.0039). The v0.2 wheel ships the fence: +# fence_view_async_shared() before the elect_one_sync mbarrier_arrive in the shipped headers' +# ll_ep.cuh combine recv pipeline — exactly the documented restore condition for this clamp. +# The T=256 rung is back on the ladder; the correctness oracle re-verifies it on every run. _LL_BUFFER_CAP = 256 -_LL_LADDER_CAP = 128 +_LL_LADDER_CAP = _LL_BUFFER_CAP class NCCLEPBackend(EPBackend): @@ -98,8 +93,10 @@ class NCCLEPBackend(EPBackend): # "-routed" marks the generation whose timed HT dispatch charges the per-step # ncclEpUpdateHandle (see dispatch()); earlier "nccl-ep-ht" rows excluded it and the # docs record that those cannot be separated by any other field — this suffix is the - # per-row discriminator that change lacked. - kernel_generation = "nccl-ep-ht-routed" + # per-row discriminator that change lacked. "v02" marks the nccl-extensions v0.2 mover + # (new kernels: LL combine fence, B200 EP16 fix, HT gains) so pre-upgrade rows never + # pool with post-upgrade rows. + kernel_generation = "nccl-ep-v02-ht-routed" SUPPORTED_MODES = ("normal", "low-latency") SUPPORTED_PRECISIONS = ("bf16",) stage_device_work = False @@ -125,7 +122,7 @@ def __init__(self, args, rank, world_size, local_rank, device): # LL decode kernels apply the top-k gate at the source (weighted), not an # unweighted rank sum — the benchmark stages the UNWEIGHTED per-expert transform # and the kernel multiplies by the gate. Same contract as deepep-v2 low-latency. - self.kernel_generation = "nccl-ep-ll" + self.kernel_generation = "nccl-ep-v02-ll" self.receive_layout = "token-expert" self.combine_weight_semantics = "weighted-kernel-sum" # NCCL EP's handle is explicitly reusable across dispatch/combine cycles (ep_test.py @@ -149,9 +146,9 @@ def __init__(self, args, rank, world_size, local_rank, device): def buffer_cap(self, args): if self._ll: - # Bounds which token counts are MEASURED. Below _LL_BUFFER_CAP today because the - # combine recv pipeline races (see the constants above); the harness reports every - # dropped rung rather than silently truncating. + # Bounds which token counts are MEASURED. Equal to _LL_BUFFER_CAP under v0.2 (the + # combine recv fence shipped — see the constants above); the two names stay separate + # so a future defect can clamp the ladder without moving the footprint. return _LL_LADDER_CAP return None diff --git a/experimental/CollectiveX/configs/platform_config.json b/experimental/CollectiveX/configs/platform_config.json index 47a0ffe98c..c5c79ec225 100644 --- a/experimental/CollectiveX/configs/platform_config.json +++ b/experimental/CollectiveX/configs/platform_config.json @@ -11,7 +11,7 @@ "scale_up_transport": "nvlink", "launcher": "single-slurm", "backends": {"deepep-v2": [8, 16], "uccl-ep": [8], "nccl-ep": [8]}, - "ll_backends": {"deepep-v2": [8], "uccl-ep": [8]}, + "ll_backends": {"deepep-v2": [8], "uccl-ep": [8], "nccl-ep": [8]}, "fabric": {"nic": "ConnectX-7 2x200GbE", "switch": "Arista 7060DX5-64S (Tomahawk4, 25.6T)"}, "operator": { "partition": "hpc-gpu-1", @@ -35,7 +35,7 @@ "scale_up_transport": "nvlink", "launcher": "single-slurm", "backends": {"deepep-v2": [8, 16], "uccl-ep": [8], "nccl-ep": [8]}, - "ll_backends": {"deepep-v2": [8], "uccl-ep": [8]}, + "ll_backends": {"deepep-v2": [8], "uccl-ep": [8], "nccl-ep": [8]}, "fabric": {"nic": "ConnectX-7 400G", "switch": "NVIDIA Quantum-2 QM9790 (25.6T, InfiniBand)"}, "operator": { "partition": "main", @@ -54,8 +54,8 @@ "scale_up_domain": 8, "scale_up_transport": "nvlink", "launcher": "single-slurm", - "backends": {"deepep-v2": [8, 16], "uccl-ep": [8, 16], "nccl-ep": [8]}, - "ll_backends": {"deepep-v2": [8, 16], "uccl-ep": [8]}, + "backends": {"deepep-v2": [8, 16], "uccl-ep": [8, 16], "nccl-ep": [8, 16]}, + "ll_backends": {"deepep-v2": [8, 16], "uccl-ep": [8], "nccl-ep": [8, 16]}, "fabric": {"nic": "8x ConnectX-7-class 400G InfiniBand rails (bare metal, gdrdrv)", "switch": "InfiniBand (model not surveyed)"}, "operator": { "partition": "batch_1", @@ -77,7 +77,7 @@ "scale_up_transport": "nvlink", "launcher": "single-slurm", "backends": {"deepep-v2": [8, 16], "nccl-ep": [8]}, - "ll_backends": {"deepep-v2": [8]}, + "ll_backends": {"deepep-v2": [8], "nccl-ep": [8]}, "fabric": {"nic": "ConnectX-8 2x400GbE", "switch": "NVIDIA Spectrum-X SN5600 (51.2T)"}, "operator": { "partition": "batch_1", @@ -104,7 +104,7 @@ "scale_up_transport": "mnnvl", "launcher": "gb-nv", "backends": {"deepep-v2": [8, 16], "nccl-ep": [8, 16], "flashinfer-ep": [8, 16]}, - "ll_backends": {"deepep-v2": [8, 16]}, + "ll_backends": {"deepep-v2": [8, 16], "nccl-ep": [8, 16]}, "fabric": {"nic": "MNNVL (scale-out not used)", "switch": "NVLink NVL72"}, "operator": { "partition": "batch", @@ -124,7 +124,7 @@ "scale_up_transport": "mnnvl", "launcher": "gb-nv", "backends": {"deepep-v2": [8, 16], "nccl-ep": [8, 16], "flashinfer-ep": [8, 16]}, - "ll_backends": {"deepep-v2": [8, 16]}, + "ll_backends": {"deepep-v2": [8, 16], "nccl-ep": [8, 16]}, "fabric": {"nic": "MNNVL (scale-out not used)", "switch": "NVLink NVL72"}, "operator": { "partition": "batch_1", diff --git a/experimental/CollectiveX/docs/methodology.md b/experimental/CollectiveX/docs/methodology.md index 34ebf39080..e7ecbe844d 100644 --- a/experimental/CollectiveX/docs/methodology.md +++ b/experimental/CollectiveX/docs/methodology.md @@ -183,13 +183,16 @@ than by precedent. UCCL-EP is a drop-in, API-identical DeepEP replacement that k `libibverbs`, with no NVSHMEM/IBGDA and with software message ordering, atomics, and flow control. Its scale-up is single-node `cudaIpc` over NVLink/XGMI (so the scale-up domain is one physical node, never MNNVL) and its EP16 scale-out uses the same per-SKU RDMA rails as the other backends. NCCL EP -is NVIDIA's native MoE dispatch/combine on the NCCL Device API, driven through the `nccl4py` -bindings. `normal` mode selects its `HIGH_THROUGHPUT` algorithm, whose FLAT `[N, hidden]` receive and +is NVIDIA's native MoE dispatch/combine on the NCCL Device API, shipped since v0.2 as the +`nccl-extensions` wheel (which owns `nccl.ep`; `nccl4py`, pinned alongside, provides `nccl.core`). +`normal` mode selects its `HIGH_THROUGHPUT` algorithm, whose FLAT `[N, hidden]` receive and unweighted rank-sum combine match `layout-and-dispatch-v1` exactly, so the same oracle applies. It is -NVIDIA-only and CUDA 13 only, and runs EP8 scale-up on H100/H200/B200/B300 plus EP8 and EP16 on -GB200/GB300, where EP16 stays inside the MNNVL scale-up domain. X86 EP16 scale-out is an unsupported -coverage row, its cross-node GIN path faulting inside `nccl_ep.cc` identically on RoCE and IB across -four SKUs. This is a GDAKI limit, not a fabric-selection one. FlashInfer EP is TensorRT-LLM's one-sided MNNVL `MoeAlltoAll`, in which each rank writes tokens directly into its peers' workspace windows and combine reads them back, so there is no send/recv pairing and no NVSHMEM. It is GB200/GB300-only for that reason, and runs EP8 and EP16 inside the MNNVL scale-up domain. Its combine is the one place a backend's accumulator precision changes the expectation rather than the tolerance: through 0.6.15 the kernel holds its top-k accumulators in the payload dtype and reduces them with a hand-unrolled pairwise tree, so every level rounds to BF16, and the oracle reproduces that tree exactly rather than loosening the gate to absorb it (0.6.16 rewrote the accumulator to FP32. The adapter reads the installed version and picks the matching model). Those throughput kernels run across the full token ladder in the `normal` mode. Its FP8 dispatch is the one (backend, precision) pair here that is realizable but off every deployed path. vLLM accepts only nvfp4/mxfp8/bf16 on this transport, so `sweep_matrix.py`'s `OFF_PATH_PRECISIONS` keeps it out of the default matrix and a production sweep measures only configurations an engine can select. Naming the precision explicitly (`--precisions fp8`) opts it back in for transport comparison against DeepEP V2/UCCL-EP at matching bytes and block size: the one place a precision filter ADDS rows rather than only removing them. +NVIDIA-only and CUDA 13 only, and runs EP8 scale-up on H100/H200/B300, EP8 and EP16 on B200 (v0.2 +fixes the RDMA EP16 GIN fault that pool hit under v0.1), and EP8 and EP16 on GB200/GB300, where EP16 +stays inside the MNNVL scale-up domain. X86 EP16 scale-out elsewhere (H100/H200/B300) remains an +unsupported coverage row pending revalidation: under v0.1 the cross-node GIN path faulted inside +`nccl_ep.cc` identically on RoCE and IB — a GDAKI limit, not a fabric-selection one — and only the +B200 configuration is claimed fixed in v0.2. FlashInfer EP is TensorRT-LLM's one-sided MNNVL `MoeAlltoAll`, in which each rank writes tokens directly into its peers' workspace windows and combine reads them back, so there is no send/recv pairing and no NVSHMEM. It is GB200/GB300-only for that reason, and runs EP8 and EP16 inside the MNNVL scale-up domain. Its combine is the one place a backend's accumulator precision changes the expectation rather than the tolerance: through 0.6.15 the kernel holds its top-k accumulators in the payload dtype and reduces them with a hand-unrolled pairwise tree, so every level rounds to BF16, and the oracle reproduces that tree exactly rather than loosening the gate to absorb it (0.6.16 rewrote the accumulator to FP32. The adapter reads the installed version and picks the matching model). Those throughput kernels run across the full token ladder in the `normal` mode. Its FP8 dispatch is the one (backend, precision) pair here that is realizable but off every deployed path. vLLM accepts only nvfp4/mxfp8/bf16 on this transport, so `sweep_matrix.py`'s `OFF_PATH_PRECISIONS` keeps it out of the default matrix and a production sweep measures only configurations an engine can select. Naming the precision explicitly (`--precisions fp8`) opts it back in for transport comparison against DeepEP V2/UCCL-EP at matching bytes and block size: the one place a precision filter ADDS rows rather than only removing them. A second `low-latency` mode adds each backend's decode-optimized kernel family. On DeepEP it drives the legacy `deep_ep.Buffer` low-latency decode kernels (`low_latency_dispatch`/`low_latency_combine`), @@ -211,16 +214,19 @@ EP8 on MI300X/MI325X/MI355X, and UCCL-EP EP8 on H100/H200/B200 only (the legacy `Buffer` low-latency kernels, which at EP8 run `cudaIpc` over NVLink, not the CPU-proxy RDMA path, because the adapter passes `is_intranode` and UCCL then never starts its proxies. The AMD SKUs drop LL: upstream raised `kNumMaxTopK` 9 -> 16 six days before our pin, and the resulting host assert -cannot hold on AMD's 16 warp groups). NCCL EP low-latency is HELD on every SKU. Its -`LOW_LATENCY` algorithm is the DeepEP-derived decode path, EXPERT_MAJOR receive with a source-side -weighted-kernel-sum combine — but its combine recv pipeline is a port of DeepEP's PRE-FIX code, -missing the `fence.proxy.async.shared::cta` DeepEP added in #642, and the shared-memory race that -fence closes is present on every rung, not only the T=256 rung where it was observed (1-in-5, -bimodal error 0.47 vs 0.0039 on gb300 EP8). The earlier T<=128 ladder clamp reduced exposure; it -was never a safety boundary, so a green clamped row is not publication-valid and the rows are held -in the registry until a fenced wheel ships. (Historically those rows also wedged on stale peer -signals, [NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303), fixed by the single-handle -adapter.) B300 therefore publishes no low-latency coverage beyond DeepEP V2 EP8. +cannot hold on AMD's 16 warp groups), and NCCL EP at EP8 on H100/H200/B300 and at EP8 and EP16 on +B200/GB200/GB300. NCCL EP's `LOW_LATENCY` algorithm is the DeepEP-derived decode path, EXPERT_MAJOR +receive with a source-side weighted-kernel-sum combine. These rows were held while v0.1's combine +recv pipeline — a port of DeepEP's PRE-FIX code, missing the `fence.proxy.async.shared::cta` DeepEP +added in #642 — raced on every rung (observed 1-in-5 bimodal corruption at T=256 on gb300 EP8, +error 0.47 vs 0.0039; the interim T<=128 ladder clamp reduced exposure but was never a safety +boundary, so even green clamped rows were not publication-valid). The v0.2 `nccl-extensions` wheel +ships that fence (`fence_view_async_shared` before the `emptyBarriers` arrive in `ll_ep.cuh`), +which is the documented restore condition: the ladder runs unclamped to the full receive and the +rows are enabled, subject as always to the correctness oracle. (Historically those rows also wedged +on stale peer signals, [NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303), fixed by the +single-handle adapter.) B300's only low-latency coverage beyond DeepEP V2 EP8 is this `candidate` +NCCL EP row. Whether a given SKU/backend/EP/mode cell is attempted is a capability fact. Whether it succeeded is decided only by the emitted artifact. @@ -398,8 +404,10 @@ it (as NVIDIA's own `ep_bench` does: CUDA events around dispatch and combine onl outside the loop) on the argument that its capacity-proportional cost would import a ladder-max term into dispatch; that argument describes exactly what production pays, since engines size the handle to their max token capacity and update it per step. The timed window now includes the -update; rows carry `kernel_generation` `nccl-ep-ht-routed`, and pre-change `nccl-ep-ht` rows are a -different measurement contract — the discriminator the earlier NCCL changes lacked. Low-latency +update; rows carry `kernel_generation` `nccl-ep-v02-ht-routed` (`nccl-ep-v02-ll` in low-latency +mode) — the `v02` component discriminates the `nccl-extensions` v0.2 mover from earlier wheels, and +pre-change `nccl-ep-ht`/`nccl-ep-ht-routed` rows are a different measurement contract or mover — +the per-row discriminator the earliest NCCL changes lacked. Low-latency mode has nothing to include: `ncclEpUpdateHandle` returns immediately there and the kernel reads the cached routing inside the timed dispatch. The other backends already carried this cost -- uccl-ep calls `get_dispatch_layout` inside dispatch, while deepep-v2, MoRI and FlashInfer pass diff --git a/experimental/CollectiveX/runtime/common.sh b/experimental/CollectiveX/runtime/common.sh index 26b88848c9..90dd823df4 100644 --- a/experimental/CollectiveX/runtime/common.sh +++ b/experimental/CollectiveX/runtime/common.sh @@ -48,15 +48,18 @@ COLLX_DEEPEP_V2_BUILD_GEN="dlarch1" COLLX_UCCL_REPO="https://github.com/uccl-project/uccl" COLLX_UCCL_COMMIT="fc1b582031221645ea9fce58aeb57187713145e3" -# NCCL EP (NVIDIA's native MoE dispatch/combine on the NCCL Device API). Primary path is the -# published nccl4py wheel — it bundles libnccl_ep.so's JIT runtime and pulls the matching -# nvidia-nccl-cu13 (>= 2.30, carrying the Device API + GIN nccl.ep needs). The from-source pins -# below are the fallback, deferred until on-metal bring-up shows the wheel is insufficient: -# contrib/nccl_ep is absent from the v2.29.x / v2.30.4 release tags, so any such build must use -# this post-merge master commit (which contains contrib/nccl_ep), NOT a release tag. -COLLX_NCCL4PY_SPEC="nccl4py[cu13]==0.3.1" -COLLX_NCCL_EP_REPO="https://github.com/NVIDIA/nccl" -COLLX_NCCL_EP_COMMIT="9d22d5dfec8391ee65b56df139d471f8e08e921e" +# NCCL EP v0.2 (NVIDIA's native MoE dispatch/combine on the NCCL Device API). Shipped as the +# nccl-extensions package (github.com/NVIDIA/nccl-extensions): it owns the nccl.ep module +# (libnccl_ep.so JIT runtime + bindings) — nccl4py stopped bundling nccl/ep at 0.4 — and pulls +# nccl4py for nccl.core plus nvidia-nccl-cu13==2.30.7 (Device API + GIN). nccl4py is pinned +# alongside so a cache rebuild resolves the same tree instead of whatever pip picks that day. +# v0.2 ships the combine-recv fence the v0.1 port of DeepEP lacked (the #642 analogue: +# fence_view_async_shared before mbarrier_arrive(emptyBarriers) in ll_ep.cuh — present in the +# published wheel's headers), which is what releases the low-latency ladder clamp, and fixes +# the x86 EP16 GIN fault on B200. Two whitespace-separated pip specs: the install site +# word-splits this deliberately, and the whole string keys the shared cache dir, so this +# change forces a reinstall. +COLLX_NCCL_EP_SPEC="nccl-extensions[cu13]==0.1.0 nccl4py[cu13]==0.5.0" # Print bounded command output without maintaining a parallel failure taxonomy. collx_log_tail() { diff --git a/experimental/CollectiveX/runtime/prepare_backend.sh b/experimental/CollectiveX/runtime/prepare_backend.sh index eaf453cd15..61e9ed98e3 100644 --- a/experimental/CollectiveX/runtime/prepare_backend.sh +++ b/experimental/CollectiveX/runtime/prepare_backend.sh @@ -423,9 +423,9 @@ uccl_prepare() { # ---- NCCL EP lifecycle ------------------------------------------------------ -# Slug of the pinned pip spec, safe as a cache-dir path component. +# Slug of the pinned pip specs, safe as a cache-dir path component. nccl_ep_spec_slug() { - printf '%s' "$COLLX_NCCL4PY_SPEC" | tr -cs 'A-Za-z0-9_.-' '-' + printf '%s' "$COLLX_NCCL_EP_SPEC" | tr -cs 'A-Za-z0-9_.-' '-' } # Cache root keyed by cpu + build arch + image + pinned wheel spec, under the shared /cx-cache @@ -469,18 +469,26 @@ nccl_ep_probe() { # import torch FIRST so libc10/libnccl are resident before the nccl.ep extension dlopens; then # nccl.core (libnccl.so) and nccl.ep (libnccl_ep.so JIT runtime). nccl.ep.__init__ runs its own # libnccl/libnccl_ep CUDA-major consistency check on import and raises ImportError on mismatch. + # The version line pins down WHICH libnccl_ep.so actually loaded — the one truth that matters + # when a stale cache or an image-bundled copy shadows the pinned wheel. python3 - <<'PY' +import sys + import torch # noqa: F401 import nccl.core # noqa: F401 -import nccl.ep # noqa: F401 +import nccl.ep + +print( + f"nccl.ep: libnccl_ep {nccl.ep.get_lib_version()} at {nccl.ep.get_lib_path()}", + file=sys.stderr, +) PY } -# Primary install: the published nccl4py[cu13] wheel + deps into $root/site via pip --target -# (self-contained; the runtime imports it through PYTHONPATH, so cache-hit and cache-miss paths -# import identically — mirrors uccl_install's copy-to-cache scheme). The from-source fallback -# (OpenMPI + build NCCL + contrib/nccl_ep from COLLX_NCCL_EP_COMMIT, with a matching launcher -# source-staging arm) is deferred until bring-up shows the wheel does not ship libnccl_ep.so. +# Primary install: the published nccl-extensions[cu13] wheel (owner of nccl.ep) plus the +# pinned nccl4py (nccl.core) into $root/site via pip --target (self-contained; the runtime +# imports it through PYTHONPATH, so cache-hit and cache-miss paths import identically — +# mirrors uccl_install's copy-to-cache scheme). nccl_ep_install() { local root="$1" site="$1/site" if [ -e "$root" ] || [ -L "$root" ]; then @@ -488,12 +496,15 @@ nccl_ep_install() { fi mkdir -m 700 "$root" || { collx_log "ERROR: NCCL EP cache-create failed"; return 1; } mkdir -p "$site" || { collx_log "ERROR: NCCL EP cache-site-create failed"; return 1; } - collx_log "NCCL EP: installing $COLLX_NCCL4PY_SPEC (pip --target)" + collx_log "NCCL EP: installing $COLLX_NCCL_EP_SPEC (pip --target)" # --target installs into an isolated tree and does not touch the system env, so PEP 668 does # not apply; torch is imported from the image at runtime (nccl.ep's torch interop resolver). + # $COLLX_NCCL_EP_SPEC is unquoted ON PURPOSE: it carries two whitespace-separated pip specs + # (nccl-extensions + the pinned nccl4py it would otherwise resolve unpinned). + # shellcheck disable=SC2086 python3 -m pip install -q --disable-pip-version-check --no-input \ - --target "$site" "$COLLX_NCCL4PY_SPEC" >&2 2>&1 \ - || { collx_log "ERROR: NCCL EP nccl4py install failed"; return 1; } + --target "$site" $COLLX_NCCL_EP_SPEC >&2 2>&1 \ + || { collx_log "ERROR: NCCL EP wheel install failed"; return 1; } nccl_ep_activate "$root" \ || { collx_log "ERROR: NCCL EP environment activation failed"; return 1; } nccl_ep_probe || { collx_log "ERROR: NCCL EP import probe failed"; return 1; } @@ -512,7 +523,7 @@ nccl_ep_prepare() { command -v flock >/dev/null \ || { collx_log "ERROR: flock is required for NCCL EP caching"; return 1; } mkdir -p "${root%/*}" || return 1 - collx_log "NCCL EP: preparing $COLLX_NCCL4PY_SPEC (shared cache $root)" + collx_log "NCCL EP: preparing $COLLX_NCCL_EP_SPEC (shared cache $root)" if ! ( [ ! -L "$lock_path" ] || { collx_log "ERROR: NCCL EP cache lock is unsafe"; exit 1; } (umask 077; : >> "$lock_path") && chmod 600 "$lock_path" \ @@ -527,14 +538,14 @@ nccl_ep_prepare() { fi else root="/tmp/collectivex-nccl-ep-cache-$(nccl_ep_spec_slug)" - collx_log "NCCL EP: preparing $COLLX_NCCL4PY_SPEC (node-local $root; no shared cache mounted)" + collx_log "NCCL EP: preparing $COLLX_NCCL_EP_SPEC (node-local $root; no shared cache mounted)" if [ ! -f "$root/.ready" ] || [ ! -d "$root/site" ]; then nccl_ep_install "$root" || return 1 fi fi nccl_ep_activate "$root" || return 1 nccl_ep_probe || { collx_log "ERROR: NCCL EP import probe failed"; return 1; } - collx_log "NCCL EP ready ($COLLX_NCCL4PY_SPEC; libnccl_ep.so JIT runtime, NCCL Device API LSA/GIN)" + collx_log "NCCL EP ready ($COLLX_NCCL_EP_SPEC; libnccl_ep.so JIT runtime, NCCL Device API LSA/GIN)" } # ---- container boundary ---------------------------------------------------- diff --git a/experimental/CollectiveX/tests/test_backends.py b/experimental/CollectiveX/tests/test_backends.py index fffc62bd1f..815cccbb2b 100644 --- a/experimental/CollectiveX/tests/test_backends.py +++ b/experimental/CollectiveX/tests/test_backends.py @@ -195,8 +195,9 @@ def test_an_unreadable_version_falls_back_to_the_rounding_model(self): self.assertFalse(gate(version), f"{version!r} must fall back to the safe model") -class NcclLowLatencyLadderClamp(unittest.TestCase): - """The measured ladder is clamped below the receive buffer around an unfixed upstream race.""" +class NcclLowLatencyLadderSizing(unittest.TestCase): + """The measured ladder and the receive buffer are separate knobs; under nccl-ep v0.2 (combine + fence shipped) the ladder is restored to the full buffer, and must never exceed it.""" def _module(self): with mock.patch.dict(sys.modules, _stub_modules()): @@ -204,11 +205,12 @@ def _module(self): import ep_nccl return importlib.reload(ep_nccl) - def test_the_ladder_is_clamped_below_the_buffer(self): - # Two separate numbers on purpose: clamping the ladder must not shrink the transport - # footprint, or the rungs that remain quietly measure a smaller receive plane. + def test_the_ladder_never_exceeds_the_buffer(self): + # Two separate numbers on purpose: a future clamp must shrink only what is MEASURED, + # never the transport footprint the remaining rungs are measured against. With the v0.2 + # fence shipped the clamp is lifted, so equality is the expected state. m = self._module() - self.assertLess(m._LL_LADDER_CAP, m._LL_BUFFER_CAP) + self.assertLessEqual(m._LL_LADDER_CAP, m._LL_BUFFER_CAP) self.assertLessEqual(m._LL_BUFFER_CAP, 511) def _backend(self, module, low_latency): @@ -240,10 +242,13 @@ def test_the_receive_is_sized_from_the_buffer_cap_not_the_ladder(self): # rather than on the shape of the source line that computes it. module = self._module() spec = types.SimpleNamespace(max_tokens_per_rank=99) - backend = self._backend(module, low_latency=True) - backend.create_buffer(spec) - self.assertEqual(backend.max_dispatch, module._LL_BUFFER_CAP) - self.assertNotEqual(backend.max_dispatch, module._LL_LADDER_CAP) + # Force the clamped configuration (ladder < buffer) so the sizing distinction is + # actually exercised — the shipping constants are equal under v0.2. + with mock.patch.object(module, "_LL_LADDER_CAP", 128): + backend = self._backend(module, low_latency=True) + backend.create_buffer(spec) + self.assertEqual(backend.max_dispatch, module._LL_BUFFER_CAP) + self.assertNotEqual(backend.max_dispatch, module._LL_LADDER_CAP) with mock.patch.object(module, "_LL_BUFFER_CAP", 512): sized = self._backend(module, low_latency=True) sized.create_buffer(spec)