Make ART a native multi-node RL runtime with Monarch

[FurtherAI]

Summary

This PR turns ART into a native distributed RL runtime rather than a single-node trainer wrapped in remote launch glue. Monarch owns process lifecycle and typed RPC, ART owns RL semantics and policy state, Megatron/NCCL owns distributed training, and vLLM owns inference execution.

Distribution remains an execution topology, not a second programming model. Existing single-node training code continues to use MegatronBackend() and PipelineTrainer; internally, it now compiles to the same one-host runtime used by multi-node jobs. Advanced users can provide explicit placement and service topology without changing the training loop.

The PR also adds production GLM-5.2 support optimized on H200 and B300, composable CP/EP/DP/PP/VPP training, multi-node inference, CUDA 13/Blackwell support, and the asynchronous data, packing, publication, and durability paths needed to keep those systems efficient.

Design Decisions

• The ART controller may run on any host. It is not tied to trainer rank 0.
• CPU rollout functions execute in distributed host-local worker processes. User code is referenced by verified import path rather than shipped as opaque pickled closures.
  • This does come with a utility to add your script to the PYTHONPATH so your rollout fn can be imported.
• Trainer ranks are ordinary ranks across hosts; TP, CP, EP, ETP, DP, PP, and VPP compose without node-specific trainer logic.
• One model service is one coordinated native vLLM deployment, which may span hosts. Independent inference replicas remain isolated on the follow-up branch rather than adding complexity to the core runtime.
• Shared storage is used only for durable checkpoints, optimizer generations, adapters, and retained artifacts. Live jobs, events, trajectories, packed batches, and health state use typed direct communication.
• Rollout owners retain trajectory records. The controller selects compact references, and packing workers fetch leased records directly from owners.
• Packing has a lookahead window of 1, moving it off the critical path.
• Also added cross-batch CP planning lookahead, moving it generally off the critical path.
• Warm Megatron ranks retain model and optimizer state. No recurring checkpoint copy, content hash, model reload, or optimizer reload remains.
• LoRA activation and durable persistence are independent asynchronous operations over immutable snapshots. Local multi-node transfer uses CPU-to-CPU NIXL through a transport-neutral manifest.
  • This is a nice optimization, especially for glm 5.2/large models, making publishing a lora fast.
• Failures are surfaced and cleanup is fail-closed; the runtime does not silently fall back to filesystem polling, NCCL Socket, naive cross-host EP, or stale policy state.
• Generalized-RL data models and TrainingProgramSpec are intentionally not prerequisites. Existing trajectories, trajectory groups, backend.train(), and pipeline training remain valid.

Public API

API	Purpose
art.init_megatron_runtime_config(...)	Configures topology, packed sequence length, snapshot capacity, compilation, and streaming offload while preserving the existing single-node call shape.
HostSpec, ClusterSpec, GpuPlacement	Describe physical hosts, CPU capacity, GPU identity, controller placement, and transport configuration.
TrainerMeshSpec, ModelServiceSpec, VllmParallelSpec	Describe trainer ranks and user-named model services without assigning algorithm-specific roles such as policy, judge, or teacher.
compile_topology(...)	Validates placement, rank ordering, parallel-world divisibility, endpoints, ports, GPU ownership, NCCL, and NIXL before model allocation.
ArtRuntime.start(...)	Attaches ART to an explicit Monarch host mesh for multi-node execution.
ArtRuntime.start_local(...)	Collapses the same runtime onto one host; this is what the default MegatronBackend() path uses.
InstalledAsyncCallable and runtime.rollout_executor(...)	Register a source-verified top-level async rollout function and distribute the actual autotuner-selected worker count across host CPU slots.
PackingRequest and PackedBatchRef	Move versioned trajectory references into immutable trainer-ready batches with explicit provenance and lease ownership.
runtime.start_trainer(...)	Starts a warm typed Megatron run using TrainerRuntimeSpec, TrainingRunSpec, job contracts, and progress/completion events.
runtime.start_model_service(...)	Starts and supervises one native single- or multi-host vLLM deployment as one health, version, update, and recovery domain.
art-monarch	Runs the same package-owned bootstrap locally, through SkyPilot, or on explicitly supplied hosts.

Normal single-node usage remains:

art.init_megatron_runtime_config(
    topology=art.MegatronTopologyConfig(),
    packed_sequence_length=128 * 1024,
)

async with MegatronBackend() as backend:
    ...

Explicit multi-node usage supplies a compiled runtime and rollout executor, then uses the same backend and PipelineTrainer APIs:

runtime = await ArtRuntime.start(host_mesh, compile_topology(...))
rollouts = runtime.rollout_executor(
    InstalledAsyncCallable.from_callable(rollout),
    target_workers=num_rollout_workers,
)

async with MegatronBackend(runtime=runtime) as backend:
    trainer = PipelineTrainer(..., rollout_executor=rollouts, backend=backend)

These APIs allow multiprocessing rollout workers, which is useful when rollouts perform CPU-heavy environment execution.

Runtime Flow

rollout CPU workers
    -> owner-local trajectory records
    -> descriptor/reference queue
    -> selection + immutable leases
    -> direct owner fetch
    -> prefix-tree packing + route replay finalization
    -> SHM or authenticated cross-host batch fanout
    -> warm Megatron ranks
    -> forward/backward + optimizer step
    -> bounded immutable snapshot pool
       -> NIXL transfer + vLLM activation
       -> asynchronous adapter/optimizer durability

Policy version, adapter generation, logprobs, rewards, timing, MoE routes, and mid-prefill policy changes remain attributable through this flow.

Implementation Map

The total branch diff is 280 files, +64,465/-17,589. Excluding tests and lockfiles, production, setup, examples, and documentation contribute +40,016/-9,631; tests contribute +17,163/-5,841.

Area	Change
src/art/distributed/	19 files, +10,221: typed topology, Monarch lifecycle, rollout execution, trajectory ownership, leased queues, packing, batch transport, NIXL adapter transfer, admission, and model-service supervision.
src/art/megatron/runtime/	16 files, +4,426/-211: typed runtime/job/event contracts, local and Monarch executors, warm trainer supervision, managed package runtime, compilation identity, publication, and recovery.
Remaining src/art/megatron/	64 files, +12,851/-5,189: backend cutover, distributed service coordination, CP/EP/HybridEP, optimizer state, asynchronous snapshots, BF16 LoRA serialization, and trainer instrumentation. The old filesystem service.py is deleted.
GLM-5.2 core and handler	12 files, +3,535: sparse MLA, indexer, CP stages, LoRA projections, model spec/state, TileLang kernel, and ART model-support integration.
training/pipeline_schedule.py	+995: PP/VPP scheduling with variable sequence lengths, executed batch size one, recomputation, CP, and route-replay integration.
ART/vLLM serving runtime	20 files, +3,955/-1,699 excluding lockfiles: vLLM 0.25.1 integration, distributed deployment lifecycle, binary MoE routes, policy spans, pooled fast metrics, and model-specific patches still required upstream.
Pipeline, autotuner, preprocessing, trajectories	14 files, +1,959/-781: bounded queue control, packing lookahead, logical/executed token accounting, and async trainer dispatch.
Release packaging and examples/multinode/	One-command CUDA 12/13 profiles, a locked content-addressed Megatron runtime, bundled HybridEP and NIXL/UCX build assets, managed etcd, and SkyPilot/local bootstrap examples.
tests/	80 files, +17,163/-5,841: runtime lifecycle, topology, data-plane, failure/recovery, publication, model correctness, numerical parity, packing, trainability, and E2E throughput coverage.

Workflow Tests

The workflow is a set of tests which run for each handler, proving things like parity with HF transformers, invariance to prefix tree packing, correct parallelism implementations, minimal train-inf mismatch, trainability and now e2e throughput. The throughput test uses a set of layers which fits a 128k packed seq on 2 gpus, cp2 ep2. vLLM is deployed with 2 gpus as well, and a synthetic workload is trained on. We assert things like a gap under 230ms p50 between consecutive fwd_bwd work, vLLM and trainer load, time to activate an adapter, trainer throughput matching expected isolated throughput (also catches recompilation issues), and overall tok/s. These ensure that the system is properly async and components are performing at peak speed.

In addition to the new stage, we redesigned how the workflow schedules itself, combined stages, and minimized imports, process startup, and repeated work. This turns a 70-90 minute run for one handler into approximately 60 minutes for all ten, with further scaling from additional GPUs.

Performance And Validation

• H200 CP8/EP8 retained 95.0% throughput when moved from one host to a 4+4 cross-host layout.
• Completion-heavy CP8/EP8 at 64K to CP16/EP16 at 128K retained 94.6% raw weak-scaling throughput.
• On two B300 training nodes, full 78-layer GLM-5.2 CP8/EP8/DP2 reached 15,713 logical tok/s and approximately 9.79% useful MFU, 12.0% faster than CP16/EP16 and 43.3% faster than the measured PP2/VPP3 topology at identical useful work.
  • With plenty of exploration, we have determined that cpN/epN is the most efficient topology in general for a model's minimum number of gpus. Single-node or multi-node.
  • We did make improvements to multi-node CP
• The selected three-node E2E layout uses 16 trainer GPUs and 8 inference GPUs. It reached score 195 and 308.19 accepted tok/s while retaining 96.4% of the 8:8 control's score per provisioned GPU.
  • Pretty impressive to scale the system throughput with gpus cleanly, requires balancing load carefully as well as good scaling from the gpu-heavy systems (vLLM and Megatron).
• Recurring trainer non-forward/backward wall time fell from 10.403s to 1.494s.
• The former 4.357s synchronous save-and-publish region became a 1.65ms enqueue plus 176.46ms immutable snapshot preparation; transfer, activation, and durability proceed asynchronously with bounded backpressure.
• Warm representative packing improved from 1.839s to 0.949s, replay finalization from 85.07ms to 0.030ms, and SHM finalization from 74.14ms to 18.41ms.
• The definitive post-main-merge workflow passed all ten required stages for all ten handlers on 24 B300 GPUs in approximately 50m18s. Sensitivity variants were intentionally excluded.
• Covered handlers are Llama 3 dense, Qwen 3 dense/MoE, Qwen 3.5 dense/MoE, Gemma 4 dense/MoE, DeepSeek V4, GLM-5.2, and GPT-OSS MoE.
• Release wheels were qualified from a fresh package install on CUDA 12/H200 with Apex fused extensions and on two CUDA 13/B300 hosts with EP2, HybridEP, NIXL, managed etcd, and vLLM. The final wheel, sdist, package-content checks, Ruff, formatting, type checking, hooks, and lock validation pass.
• Physical multi-node gates exited cleanly without residual ART, Monarch, Megatron, vLLM, NCCL, or GPU processes.

Intentional Scope

This PR does not add a second training API, Ray, generalized-RL program definitions, multiple independent inference replicas, merged-weight serving, the old NCCL weight-transfer engine, file-backed job dispatch, JSONL polling, or nested multi-node torchrun. Those omissions are deliberate: the delivered core is the smallest coherent runtime compatible with the efficiency target that provides correct multi-node rollout, inference, training, data movement, policy publication, durability, and single-node collapse.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
openpipe-art	🟢 Ready	View Preview	Aug 19, 2026, 1:32 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[bradhilton]

Codex/Claude found two potential blockers:

1. LocalBackend training breaks for Unsloth and Tinker. _resolve_grad_accumulation_sequences always calls service.resolve_global_grad_accumulation_sequences(), but neither UnslothService nor TinkerService implements the method. The first non-empty training batch raises AttributeError, including with an explicitly configured accumulation value. Restoring the previous fallback or implementing the method on both services would resolve the issue. A real-service regression test would help prevent recurrence.

2. Multi-member vLLM startup fails during NCCL preflight. The embedded preflight script imports _runtime_python_for_nccl_discovery, which the PR removes from art.vllm_runtime. Executing the child script produces an ImportError. Every model service with at least two members runs this preflight, preventing multi-node serving from starting. Restoring the helper or using the remaining runtime-resolution helpers would resolve the issue. A test executing the embedded script would cover the import.

Additional findings are listed below. Use your discretion to determine whether fixes belong in this PR or follow-up work:

• _advance_skipped_step() removes self._services[model.name], while the dictionary uses model storage tuples as keys. A failed adapter registration can leave a closed service cached.
• NIXL adapter transfers have no deadline and can occupy the single transport worker indefinitely.
• Managed etcd listens unauthenticated on 0.0.0.0. An earlier cleanup failure can skip etcd shutdown and leave the subprocess orphaned.
• import art redirects unset Hugging Face, Torch, Triton, and vLLM cache paths to shared /tmp/art-cache. Client-only usage and shared hosts may encounter unexpected downloads, permission failures, or cache-isolation problems.
• Closing an InMemoryPackedBatch leaves batch.tensors referencing unmapped shared memory. Post-close access produced a segmentation fault during reproduction, although no current production call path accessing the tensors after close was found.
• IPv6 endpoint URLs lack brackets, and multi-host model-service validation accepts a loopback leader endpoint.
• Adding numpy<2 to base dependencies can force downgrades for lightweight installations.

[FurtherAI]

@bradhilton Thanks for the detailed review. I addressed both blockers and all secondary findings in a6227e480..09ed93d1e.

Blockers

1. Restored explicit gradient-accumulation contracts on both services. Tinker accepts only its fixed value of 1; Unsloth resolves and validates trainer_args.gradient_accumulation_steps. LocalBackend can therefore keep one typed service contract without an implicit fallback.
2. Restored vLLM runtime-Python resolution using the current runtime helpers and changed the embedded NCCL probe to call it. The test executes the embedded script rather than only inspecting its source. The exact preflight also passed through the real two-host SSH launcher on B300 nodes.

Additional findings

• Fixed skipped-step eviction to use the model storage tuple key.
• Added a propagated transfer deadline to NIXL polling; timed-out handles are released.
• Managed etcd now listens only on its advertised endpoint, runs under ART's parent-death process supervisor, and is closed before other host cleanup. Cleanup continues through independent failures. It remains unauthenticated under ART's existing trusted-private-cluster boundary rather than exposing 0.0.0.0.
• Removed model-cache configuration from top-level import art. The import-boundary test rejects ART-managed cache defaults while allowing Torch's own Inductor default, which Transformers initializes internally.
• InMemoryPackedBatch.close() now revokes tensor access before unmapping shared memory; subsequent access raises rather than retaining a dangling tensor.
• IPv6 literals are bracketed, and multi-host services reject loopback leader and rendezvous endpoints.
• Base installs now use numpy>=1.26; <2 remains limited to the Megatron, Tinker, and other extras that require it. A base wheel imported successfully with NumPy 2.5.2.

Physical validation also exposed and fixed SSH workers not receiving the admitted NCCL/runtime environment. Final selective validation was:

• 61 review-related tests passed.
• 9 real CUDA/NCCL tests passed on 1, 2, and 4 B300 GPUs, including LoRA updates, optimizer restore, recomputation, and TP/DP/CP reductions.
• The exact two-host vLLM/NCCL preflight passed and left no ART, etcd, or GPU processes behind.