modelscope · loveleaves · Sep 17, 2026 · Sep 17, 2026 · Sep 17, 2026 · Sep 17, 2026
diff --git a/docs/source_en/Components/Checkpoint Engine/CheckpointEngine.md b/docs/source_en/Components/Checkpoint Engine/CheckpointEngine.md
@@ -39,7 +39,7 @@ class CheckpointEngine(ABC):
 
 ## Available Checkpoint Engines
 
-Twinkle provides two checkpoint engine implementations:
+Twinkle provides three checkpoint engine implementations:
 
 ### NCCLCheckpointEngine
 
@@ -61,10 +61,21 @@ A checkpoint engine that uses HCCL for weight transfer between Ascend NPUs.
 
 See: [HCCLCheckpointEngine](HCCLCheckpointEngine.md)
 
+### XCCLCheckpointEngine
+
+A checkpoint engine that transfers weights over BKCL (XCCL) on Kunlunxin XPU.
+
+- XPU Support: Drop-in for NCCLCheckpointEngine on the cuda-alike XPU route
+- Relay Fallback: Handles duplicate local device indices via a stateless `ProcessGroupXCCL` plus a socket relay
+- Compatible Interface: Inherits `NCCLCheckpointEngine`, reusing bucketing and metadata handshake
+
+See: [XCCLCheckpointEngine](XCCLCheckpointEngine.md)
+
 ## How to Choose
 
 - **NCCLCheckpointEngine**: Suitable for GPU environments, provides the highest transfer performance
 - **HCCLCheckpointEngine**: Suitable for Ascend NPU environments
+- **XCCLCheckpointEngine**: Suitable for Kunlunxin XPU environments (selected automatically)
 
 > Checkpoint engine is a key component of RLHF training infrastructure, ensuring that trainers and samplers use consistent model weights.
 > Currently, synchronization is divided into two cases based on merge_and_sync=True/False. When set to True, the LoRA is merged into the base model and then synchronized.

diff --git a/docs/source_en/Components/Checkpoint Engine/XCCLCheckpointEngine.md b/docs/source_en/Components/Checkpoint Engine/XCCLCheckpointEngine.md
@@ -0,0 +1,37 @@
+# XCCLCheckpointEngine
+
+A checkpoint engine for Kunlunxin XPU that transfers weights over BKCL (mounted into PyTorch as the `nccl`/XCCL backend).
+
+## Usage Example
+
+```python
+from twinkle.checkpoint_engine import XCCLCheckpointEngine
+
+engine = XCCLCheckpointEngine(bucket_size=512<<20)
+# Usage is the same as NCCLCheckpointEngine
+```
+
+On XPU this engine is selected automatically by `CheckpointEngineManager` / `CheckpointEngineMixin`, so you normally do not construct it directly.
+
+## Features
+
+- **Drop-in for NCCL**: Inherits `NCCLCheckpointEngine`; bucketing, ZMQ metadata handshake, and double buffering are reused unchanged.
+- **XCCL group + relay fallback**: Replaces the direct `ProcessGroupNCCL` construction (which deadlocks when two ranks share a local device index) with a stateless `ProcessGroupXCCL`, plus a relay path for colliding ranks.
+- **Store-based init barrier**: The readiness barrier runs through the TCPStore instead of a device collective.
+
+## Why a Separate Engine
+
+BKCL identifies a rank's device by its **local device index** and does not translate `CUDA_VISIBLE_DEVICES` to physical cards. In a separated trainer/sampler deployment on a single host, both sides see their device as local `0`, producing duplicate `(host, index)` pairs. Directly building `ProcessGroupNCCL` then fails to form the ring and the first broadcast deadlocks (600s WorkXCCL timeout).
+
+`XCCLCheckpointEngine` exchanges `(hostname, local_device_index)` through the store, keeps colliding ranks out of the XCCL group, and serves them through the lowest-rank XCCL member over direct TCP sockets (relay). Non-colliding ranks use a flat stateless `ProcessGroupXCCL`.
+
+## Use Cases
+
+- Weight synchronization (train ↔ sample) on Kunlunxin XPU
+- Colocate / separated RL (GRPO) with full-weight sync (`merge_and_sync=True`)
+
+## Limitations
+
+- Multi-host (inter/intra-node) broadcast is implemented but not yet validated; only single-host relay mode is verified.
+
+> On Kunlunxin XPU, use `merge_and_sync=True` for weight sync; LoRA sampling is currently blocked by vendor kernel dtype limitations (see the [XPU Support](../../Usage%20Guide/XPU-Support.md) guide).
diff --git a/docs/source_en/Components/Checkpoint Engine/index.rst b/docs/source_en/Components/Checkpoint Engine/index.rst
@@ -6,3 +6,4 @@ Checkpoint Engine
    CheckpointEngine.md
    NCCLCheckpointEngine.md
    HCCLCheckpointEngine.md
+   XCCLCheckpointEngine.md
diff --git a/docs/source_en/Usage Guide/Installation.md b/docs/source_en/Usage Guide/Installation.md
@@ -53,5 +53,6 @@ sh INSTALL_MEGATRON.sh
 | GPU A10/A100/H100/RTX series |                                        |
 | GPU T4/V100                 | Does not support bfloat16, Flash-Attention |
 | Ascend NPU                  | Some operators not supported            |
+| Kunlunxin XPU (P800)        | Supported via the cuda-alike route; some operators fp16-only, see [XPU Support](XPU-Support.md) |
 | PPU                         | Supported                              |
 | CPU                         | Supports partial components like dataset, dataloader |
diff --git a/docs/source_en/Usage Guide/XPU-Support.md b/docs/source_en/Usage Guide/XPU-Support.md
@@ -0,0 +1,126 @@
+# XPU (Kunlunxin) Quick Start Guide
+
+This document describes how to install and use the Twinkle framework on Kunlunxin XPU (P800 series).
+
+## How XPU Support Works
+
+Unlike Ascend NPU, Kunlunxin XPU does **not** introduce a separate device type in Twinkle. It relies on `torch_xmlir`, which rewrites `torch.cuda` symbols at import time so that:
+
+- `torch.cuda.is_available()` returns `True` and devices report as `cuda:N`
+- `libbkcl.so` (BKCL) is mounted as PyTorch's `nccl` backend (XCCL)
+- `CUDA_VISIBLE_DEVICES` is taken over by the XPU runtime
+
+Because of this "cuda-alike" route, most of Twinkle's GPU code path works unchanged. Only a few CUDA-specific assumptions that the symbol rewrite does not cover need adaptation. The `XPU` platform therefore **inherits from `GPU`** (`src/twinkle/utils/platforms/xpu.py`) rather than being a fully independent device backend like NPU.
+
+## Environment Requirements
+
+Adaptation was validated inside the vendor container image (`kylin_v11-vllm_021_swift:*`). The verified component matrix:
+
+| Component | Version | Notes |
+|---|---|---|
+| OS / Python | Kylin V11 / 3.10.14 | Meets Twinkle `>=3.10` |
+| torch | 2.9.0 | Symbol-rewritten by `torch_xmlir` |
+| vllm / vllm_kunlun | 0.21.0 / 0.21.0.dev0 (also revalidated on 0.25.1) | `KunlunPlatform: device_type="cuda", dist_backend="nccl"` |
+| megatron-core / transformers / triton | 0.16.1 / 5.9.0 / 3.5.0 | triton runs on the xmlir backend |
+| XPU kernels | xpu_flash_attn, xpu_fla, kunlun_ops, xspeedgate_ops, xformers | Provided by the vendor container |
+
+**Notes**:
+- The Kunlunxin driver, `torch_xmlir`, and the XPU-enabled `torch` / `vllm_kunlun` builds are provided by Kunlunxin (typically via the vendor container image). Twinkle does not install them.
+- `vllm_kunlun` and its dependencies come from Kunlunxin; some kernels currently have dtype restrictions (see [Known Limitations](#known-limitations)).
+
+## Supported Hardware
+
+- Kunlunxin P800 (OAM), verified on an 8-card host
+
+## Installation Steps
+
+### 1. Prepare the XPU environment
+
+Use the Kunlunxin-provided container image (or a host with the Kunlunxin driver, `torch_xmlir`, and the XPU builds of `torch` / `vllm` / `vllm_kunlun` installed). Twinkle does not manage these components.
+
+### 2. Install Twinkle
+
+Install from source with `--no-deps` so pip does not attempt to replace the vendor's `torch` / `vllm` builds:
+
+```bash
+git clone https://github.com/modelscope/twinkle.git
+cd twinkle
+pip install -e . --no-deps
+pip install pyzmq
+# If the container is missing debug/runtime helpers:
+pip install h5py prettytable func_timeout ray redis
+```
+
+### 3. Verify Installation
+
+Create test script `verify_xpu.py`:
+
+```python
+import torch
+import twinkle  # triggers ensure_xpu_compat()
+from twinkle.utils.platforms import Platform
+
+print(f"PyTorch version: {torch.__version__}")
+print(f"CUDA (XPU) available: {torch.cuda.is_available()}")
+print(f"Device count: {torch.cuda.device_count()}")
+print(f"Detected platform: {Platform.get_platform().__name__}")
+
+if torch.cuda.is_available():
+    x = torch.randn(3, 3).cuda()
+    y = torch.randn(3, 3).cuda()
+    print(f"XPU computation test passed: {(x + y).shape}")
+```
+
+Run verification:
+
+```bash
+python verify_xpu.py
+```
+
+A successful run reports `Detected platform: XPU`, `CUDA (XPU) available: True`, and the correct device count. Platform detection keys off the presence of `xpu-smi` on `PATH`.
+
+## Verified Capabilities
+
+The following have been validated end-to-end on Kunlunxin P800 (single-host, 8-card):
+
+| Capability | Backend | Status | Notes |
+|---|---|---|---|
+| FSDP2 LoRA SFT | transformers | ✅ Verified | Single-card, loss converges as on GPU |
+| vLLM sampling (TP=1) | vLLM | ✅ Verified | Requires `enforce_eager=True`; text models |
+| vLLM sampling (TP=2) | vLLM | ✅ Verified | Tensor parallel |
+| GRPO (colocate) | native_fsdp | ✅ Verified | Full-weight sync (`merge_and_sync=True`); relay ~0.7 GB/s |
+| Megatron LoRA (TP=2) | Megatron | ✅ Verified | Zero Twinkle code change; cuda-alike path |
+| Weight sync (train↔sample) | XCCL | ✅ Verified | Via `XCCLCheckpointEngine`, single-host relay |
+
+## Known Limitations
+
+These are constraints of the current Kunlunxin XPU stack (vendor kernels / `vllm_kunlun`), not of Twinkle logic:
+
+- **LoRA sampling is not usable**: `vLLMSampler` with `enable_lora=True` cannot start. The underlying LoRA kernels (`bgmv_shrink_cluster`, `sgmv_expand_sdnn`, `sgmv_expand_slice`) are fp16-only on the 0.21 stack. Use full-weight sync (`merge_and_sync=True`) for RL. (On the 0.25.1 stack the vendor added a bf16 adapter layer, but other LoRA-path issues remain.)
+- **fp16 required for some models**: GDN FLA kernels are fp16-only; Qwen3.5 inference must run in float16.
+- **CUDA graph disabled**: Use `enforce_eager=True` for vLLM; graph capture on XPU is not yet validated.
+- **Multimodal (Qwen-VL) blocked**: The ViT SDPA kernel triggers a hardware-level `kl3ChannelCheckErrors` / `noc idle timeout` that requires a physical card reset. Use text-only models.
+- **Single-host only**: `XCCLCheckpointEngine` multi-host (inter/intra-node) broadcast is implemented but not yet validated; only single-host relay mode is verified.
+- **`logprobs` on vllm_kunlun 0.25.1**: The logprobs path can return NaN/uninitialized top-logprob token ids, causing tokenizer `OverflowError`. Reported to the vendor.
+
+## Platform Internals (for reference)
+
+The XPU adaptation is concentrated in a few files:
+
+- `src/twinkle/utils/platforms/xpu.py` — `XPU` platform (inherits `GPU`); `ensure_xpu_compat()` neutralizes the native Intel-XPU stub in `torch.xpu`; a device-UUID fallback chain (`current_platform` → `xpu-smi -q` Bus Id → `xpu-smi -L` UUID → sha1) for vLLM.
+- `src/twinkle/utils/platforms/base.py` — detects XPU via `xpu-smi` on `PATH`.
+- `src/twinkle/infra/_ray/ray_helper.py` — registers the cuda-alike GPU count through `ray.init(num_gpus=N)` (Ray cannot autodetect XPUs). Remote workers started via `ray start` must pass `--num-gpus`.
+- `src/twinkle/checkpoint_engine/xpu_checkpoint_engine.py` — `XCCLCheckpointEngine` (see the Checkpoint Engine docs).
+
+## Reference Resources
+
+- [vLLM-Kunlun](https://github.com/baidu/vLLM-Kunlun)
+- [Twinkle GitHub](https://github.com/modelscope/twinkle)
+- [Twinkle Documentation](https://twinkle.readthedocs.io/)
+
+## Getting Help
+
+If you encounter issues during use:
+
+1. **Submit an Issue**: [Twinkle GitHub Issues](https://github.com/modelscope/twinkle/issues)
+2. **Vendor stack issues** (LoRA kernels, logprobs, multimodal): report to Kunlunxin / vLLM-Kunlun.
diff --git a/docs/source_en/index.rst b/docs/source_en/index.rst
@@ -13,6 +13,7 @@ Twinkle DOCUMENTATION
    Usage Guide/Installation.md
    Usage Guide/Server and Client/index.rst
    Usage Guide/NPU-Support.md
+   Usage Guide/XPU-Support.md
    Usage Guide/Train-as-a-Service.md
    Usage Guide/Agentic-RL-Deployment-and-Training.md
    Usage Guide/Agentic-Evaluator.md

diff --git a/docs/source_zh/index.rst b/docs/source_zh/index.rst
@@ -13,6 +13,7 @@ Twinkle DOCUMENTATION
    使用指引/安装.md
    使用指引/服务端和客户端/index.rst
    使用指引/NPU的支持.md
+   使用指引/XPU的支持.md
    使用指引/训练服务.md
    使用指引/Agentic RL部署与训练.md
    使用指引/Agentic评测.md

diff --git a/docs/source_zh/使用指引/XPU的支持.md b/docs/source_zh/使用指引/XPU的支持.md
@@ -0,0 +1,126 @@
+# XPU（昆仑芯）开箱指南
+
+本文档介绍如何在昆仑芯 XPU（P800 系列）环境下安装和使用 Twinkle 框架。
+
+## XPU 支持原理
+
+与昇腾 NPU 不同，昆仑芯 XPU 在 Twinkle 中**不引入独立的设备类型**，而是依赖 `torch_xmlir` 在 import 阶段对 `torch.cuda` 符号进行重写：
+
+- `torch.cuda.is_available()` 返回 `True`，设备表现为 `cuda:N`
+- `libbkcl.so`（BKCL）以 `nccl` 后端（XCCL）挂载到 PyTorch
+- `CUDA_VISIBLE_DEVICES` 由 XPU runtime 接管
+
+得益于这种 "cuda-alike" 路线，Twinkle 的 GPU 代码路径绝大部分可原生复用，仅需适配少数符号重写未覆盖的 CUDA 专属假设。因此 `XPU` 平台**继承自 `GPU`**（`src/twinkle/utils/platforms/xpu.py`），而非像 NPU 那样是完全独立的设备后端。
+
+## 环境要求
+
+适配在厂商容器镜像（`kylin_v11-vllm_021_swift:*`）内验证，已确认的组件矩阵如下：
+
+| 组件 | 版本 | 说明 |
+|---|---|---|
+| OS / Python | Kylin V11 / 3.10.14 | 满足 Twinkle `>=3.10` |
+| torch | 2.9.0 | 被 `torch_xmlir` 符号重写 |
+| vllm / vllm_kunlun | 0.21.0 / 0.21.0.dev0（0.25.1 亦复验通过） | `KunlunPlatform: device_type="cuda", dist_backend="nccl"` |
+| megatron-core / transformers / triton | 0.16.1 / 5.9.0 / 3.5.0 | triton 走 xmlir 后端 |
+| XPU kernels | xpu_flash_attn、xpu_fla、kunlun_ops、xspeedgate_ops、xformers | 由厂商容器提供 |
+
+**说明**：
+- 昆仑芯驱动、`torch_xmlir` 以及 XPU 版的 `torch` / `vllm_kunlun` 由昆仑芯提供（通常通过厂商容器镜像），Twinkle 不负责安装。
+- `vllm_kunlun` 及其依赖来自昆仑芯；部分算子当前存在 dtype 限制（见[已知限制](#已知限制)）。
+
+## 支持的硬件
+
+- 昆仑芯 P800（OAM），已在单机 8 卡环境验证
+
+## 安装步骤
+
+### 1. 准备 XPU 环境
+
+使用昆仑芯提供的容器镜像（或已安装昆仑芯驱动、`torch_xmlir` 及 XPU 版 `torch` / `vllm` / `vllm_kunlun` 的宿主机）。这些组件不由 Twinkle 管理。
+
+### 2. 安装 Twinkle
+
+从源码以 `--no-deps` 方式安装，避免 pip 覆盖厂商的 `torch` / `vllm` 构建：
+
+```bash
+git clone https://github.com/modelscope/twinkle.git
+cd twinkle
+pip install -e . --no-deps
+pip install pyzmq
+# 若容器缺少调试/运行时辅助依赖：
+pip install h5py prettytable func_timeout ray redis
+```
+
+### 3. 验证安装
+
+创建测试脚本 `verify_xpu.py`：
+
+```python
+import torch
+import twinkle  # 触发 ensure_xpu_compat()
+from twinkle.utils.platforms import Platform
+
+print(f"PyTorch version: {torch.__version__}")
+print(f"CUDA (XPU) available: {torch.cuda.is_available()}")
+print(f"Device count: {torch.cuda.device_count()}")
+print(f"Detected platform: {Platform.get_platform().__name__}")
+
+if torch.cuda.is_available():
+    x = torch.randn(3, 3).cuda()
+    y = torch.randn(3, 3).cuda()
+    print(f"XPU computation test passed: {(x + y).shape}")
+```
+
+运行验证：
+
+```bash
+python verify_xpu.py
+```
+
+运行成功时应输出 `Detected platform: XPU`、`CUDA (XPU) available: True` 以及正确的设备数量。平台检测依据 `PATH` 中是否存在 `xpu-smi`。
+
+## 已验证能力
+
+以下能力已在昆仑芯 P800（单机 8 卡）上端到端验证：
+
+| 能力 | 后端 | 状态 | 说明 |
+|---|---|---|---|
+| FSDP2 LoRA SFT | transformers | ✅ 已验证 | 单卡，loss 收敛与 GPU 一致 |
+| vLLM 采样（TP=1） | vLLM | ✅ 已验证 | 需 `enforce_eager=True`；纯文本模型 |
+| vLLM 采样（TP=2） | vLLM | ✅ 已验证 | 张量并行 |
+| GRPO（colocate） | native_fsdp | ✅ 已验证 | 全量权重同步（`merge_and_sync=True`）；relay ~0.7 GB/s |
+| Megatron LoRA（TP=2） | Megatron | ✅ 已验证 | Twinkle 侧零改动；cuda-alike 路径 |
+| 权重同步（训练↔采样） | XCCL | ✅ 已验证 | 经 `XCCLCheckpointEngine`，单机 relay |
+
+## 已知限制
+
+以下为当前昆仑芯 XPU 栈（厂商 kernel / `vllm_kunlun`）的限制，而非 Twinkle 逻辑问题：
+
+- **LoRA 采样不可用**：`enable_lora=True` 的 `vLLMSampler` 无法启动。底层 LoRA 算子（`bgmv_shrink_cluster`、`sgmv_expand_sdnn`、`sgmv_expand_slice`）在 0.21 栈上仅支持 fp16。RL 请使用全量权重同步（`merge_and_sync=True`）。（0.25.1 栈厂商新增了 bf16 适配层，但 LoRA 路径仍有其他问题。）
+- **部分模型需 fp16**：GDN FLA 算子仅支持 fp16，Qwen3.5 推理必须用 float16。
+- **CUDA graph 关闭**：vLLM 需 `enforce_eager=True`；XPU 上的 graph 捕获尚未验证。
+- **多模态（Qwen-VL）阻塞**：ViT SDPA 算子触发硬件级 `kl3ChannelCheckErrors` / `noc idle timeout`，需物理重置卡。请使用纯文本模型。
+- **仅支持单机**：`XCCLCheckpointEngine` 的多机（节点间/节点内）广播已实现但尚未验证，当前仅验证单机 relay 模式。
+- **vllm_kunlun 0.25.1 的 `logprobs`**：logprobs 路径可能返回 NaN/未初始化的 top-logprob token id，导致 tokenizer `OverflowError`。已反馈厂商。
+
+## 平台内部实现（参考）
+
+XPU 适配集中在少数文件：
+
+- `src/twinkle/utils/platforms/xpu.py` — `XPU` 平台（继承 `GPU`）；`ensure_xpu_compat()` 桩化 `torch.xpu` 中的原生 Intel-XPU 桩；为 vLLM 提供设备 UUID 降级链（`current_platform` → `xpu-smi -q` Bus Id → `xpu-smi -L` UUID → sha1）。
+- `src/twinkle/utils/platforms/base.py` — 通过 `PATH` 中的 `xpu-smi` 检测 XPU。
+- `src/twinkle/infra/_ray/ray_helper.py` — 经 `ray.init(num_gpus=N)` 注册 cuda-alike 的 GPU 数量（Ray 无法自动发现 XPU）。经 `ray start` 启动的远程 worker 需手动传 `--num-gpus`。
+- `src/twinkle/checkpoint_engine/xpu_checkpoint_engine.py` — `XCCLCheckpointEngine`（详见检查点引擎文档）。
+
+## 参考资源
+
+- [vLLM-Kunlun](https://github.com/baidu/vLLM-Kunlun)
+- [Twinkle GitHub](https://github.com/modelscope/twinkle)
+- [Twinkle 文档](https://twinkle.readthedocs.io/)
+
+## 获取帮助
+
+如果您在使用过程中遇到问题：
+
+1. **提交 Issue**：[Twinkle GitHub Issues](https://github.com/modelscope/twinkle/issues)
+2. **厂商栈问题**（LoRA 算子、logprobs、多模态）：请反馈昆仑芯 / vLLM-Kunlun。
diff --git a/docs/source_zh/使用指引/安装.md b/docs/source_zh/使用指引/安装.md
@@ -53,5 +53,6 @@ sh INSTALL_MEGATRON.sh
 | GPU A10/A100/H100/RTX系列等 |                             |
 | GPU T4/V100等             | 不支持bfloat16、Flash-Attention |
 | Ascend NPU               | 部分算子不支持                     |
+| 昆仑芯 XPU（P800）         | 经 cuda-alike 路线支持；部分算子仅 fp16，详见 [XPU 的支持](XPU的支持.md) |
 | PPU                      | 支持                          |
 | CPU                      | 支持dataset、dataloader等部分组件   |
diff --git a/docs/source_zh/组件/检查点引擎/CheckpointEngine.md b/docs/source_zh/组件/检查点引擎/CheckpointEngine.md
@@ -39,7 +39,7 @@ class CheckpointEngine(ABC):
 
 ## 可用的检查点引擎
 
-Twinkle 提供了两种检查点引擎实现:
+Twinkle 提供了三种检查点引擎实现:
 
 ### NCCLCheckpointEngine
 
@@ -61,10 +61,21 @@ Twinkle 提供了两种检查点引擎实现:
 
 详见: [HCCLCheckpointEngine](HCCLCheckpointEngine.md)
 
+### XCCLCheckpointEngine
+
+使用 BKCL（XCCL）在昆仑芯 XPU 上进行权重传输的检查点引擎。
+
+- XPU 支持: 在 cuda-alike XPU 路线上作为 NCCLCheckpointEngine 的直替
+- relay 兜底: 通过 stateless `ProcessGroupXCCL` 加 socket 中继处理重复的 local device index
+- 兼容接口: 继承 `NCCLCheckpointEngine`，复用分桶与元数据握手
+
+详见: [XCCLCheckpointEngine](XCCLCheckpointEngine.md)
+
 ## 如何选择
 
 - **NCCLCheckpointEngine**: 适用于 GPU 环境,提供最高的传输性能
 - **HCCLCheckpointEngine**: 适用于昇腾 NPU 环境
+- **XCCLCheckpointEngine**: 适用于昆仑芯 XPU 环境（自动选择）
 
 > 检查点引擎是 RLHF 训练基础设施的关键组件,确保训练器和采样器使用一致的模型权重。
 > 目前的同步分为merge_and_sync=True/False两种情况，为True时将lora合并仅基模并同步，为False时仅同步lora权重。另外，多租户直接附加lora文件到vLLM上，在merge_and_sync=False，或使用多租户时，