Building cross-hardware and multi-model RL post-training infrastructure for kernel-level train–inference consistency.
Architecture · Current scope · Results · Hardware support · Quick start · Documentation
RL-Kernel is high-performance infrastructure for RL post-training. It provides deterministic operators for consistent numerical computation across rollout and training engines, together with hardware-specific kernels for faster execution and lower memory use in GRPO, PPO, and related workloads.
Today, the end-to-end path covers Qwen3-8B Dense with vime, vLLM, and Megatron-LM. Work on DeepSeek-V4 Flash MoE, Miles, and AReaL is ongoing.
Rollout and training engines can produce different log probabilities for the same tokens and model weights because their kernels, batching, and reduction orders differ. Those differences enter the policy ratios and KL terms used by RL algorithms.
- Exact train–inference consistency: deterministic operators keep rollout and training computations aligned. The published experiment records exact runtime LogP agreement across all 200 training steps.
- RL operators: deterministic attention, dense FFN, LogP, GRPO and PPO objectives, and collectives cover the numerical boundaries in RL post-training.
- Performance: fused computation and hardware-specific kernels reduce rollout time, memory use, and synchronization costs.
- vime integration: vime orchestrates vLLM rollout and Megatron-LM training, with RL-Kernel supplying the operators used by both engines.
- Hardware: NVIDIA SM90 and AMD gfx942 are supported. Ascend dav_c220 has partial operator coverage. Support for other hardware is in progress.
RL-Kernel sits between execution engines and accelerator backends. Its runtime adapters select the operator implementation for each backend while keeping the same numerical contract across rollout and training.
The architecture below shows how orchestration frameworks, execution engines, RL-Kernel operators, and hardware backends fit together.
The current end-to-end path uses Qwen3-8B Dense with vime.
| Area | Current | Next |
|---|---|---|
| Model | Qwen3-8B Dense | DeepSeek-V4-Flash-0731 MoE |
| Orchestration | vime | Miles and AReaL |
| Engines | vLLM rollout and Megatron-LM training | More rollout and training engines |
RL-Kernel currently supports the following hardware targets.
| Hardware | Architecture | Software | Status |
|---|---|---|---|
| NVIDIA H100, H200, GH200 | SM90 | CUDA | Supported |
| AMD Instinct MI300A, MI300X, MI325X | gfx942 | ROCm | Supported |
| Huawei Ascend dav_c220 | dav-2201 | CANN 9.1.0 and Ascend C | Partial |
| Moore Threads | In development | MUSA | In progress |
The published end-to-end benchmark was run on H100. The ROCm extension and backend checks were verified on MI300X. Ascend support is limited to dav_c220. Support for other hardware models is in progress.
Use a compatible vime environment on an eight-GPU H100 or MI300X node. Clone the project and follow the installation guide for your CUDA or ROCm build:
git clone https://github.com/RL-Align/RL-Kernel.git
cd RL-Kernel
Complete the one-time CUDA setup or ROCm setup. Save the backend, Python, framework, model and data paths in .rlk-profile.json or select a profile with RLK_REPRO_PROFILE. No launcher edits are needed. Both backends then use the same command:
./rlk run --tp 4 --rollout-tp 4 --temperature 0.7 --top-p 0.95 \
--lr 5e-7 --kl-coef 0.01 --max-response-len 6912 --max-tokens-per-gpu 4096 --steps 200Training TP and rollout TP are independent: choose 1, 2, 4 or 8. Training CP defaults to 8 / TP; set --cp explicitly if needed. run waits, validates train/rollout LogP, and defaults to consistency mode without rollout-logprob reuse. Add --mode native for a native comparison, or replace run with plan to inspect the command without launching a job.
On ROCm, this command selects Triton chunked Attention and sparse top-p
logprob/monitoring-entropy scoring. Training logprobs are independently
recomputed and validation requires bitwise agreement with rollout. Apply the
updated ROCm companion patches and rebuild the extension when updating.
With the supplied ROCm profile, a three-step check measured 58.29 s/step
versus native's 57.40 s (+1.55%), with zero raw-bit logprob mismatches.
Use --steps 3 for that short check; timing depends on generated lengths and
the environment, and the native execution-record limitation is documented below.
See ROCm performance reproduction
for the full comparison command, measurements and validation limits.
On CUDA, rollout CP and top-k are configurable too; this short check performs two real updates and validates their artifacts:
./rlk verify --tp 1 --rollout-tp 1 --rollout-cp 8 --temperature 0.7 --top-p 0.95 --top-k -1Rollout TP × CP must divide eight. CUDA accepts top-k -1 (disabled) or a positive integer, and temperature 0 for greedy sampling. ROCm currently requires top-k -1 and positive temperature. ROCm rollout CP > 1 remains unvalidated, and verify is CUDA-only. See the CUDA/ROCm support table and measured H100 results for exact coverage.
Join us on Slack or WeChat, and open an issue for bugs and feature requests. Contributions to kernels, framework integrations, hardware adaptation, and benchmarks are welcome. See the contributing guide.
RL-Kernel builds on the work of the open-source AI infrastructure community, including vime, vLLM, Megatron-LM, and FlashInfer. We thank their contributors and everyone helping bring RL-Kernel to new accelerators.
Licensed under the Apache License 2.0.