Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
c3a2a1a
merge
wangxingjun778 Oct 13, 2026
950b941
ok Merge branch 'feat/mhs' of github.com:modelscope/leapflow into fea…
wangxingjun778 Oct 15, 2026
acd9464
ok Merge branch 'feat/mhs' of github.com:modelscope/leapflow into fea…
wangxingjun778 Oct 15, 2026
78a2563
remove unused content
wangxingjun778 Oct 16, 2026
6a5aea3
refactor and enhance hardware framework
wangxingjun778 Oct 19, 2026
e5b133d
add enhance hardware skills
wangxingjun778 Oct 20, 2026
1093d18
ok Merge branch 'feat/mhs' of github.com:modelscope/leapflow into fea…
wangxingjun778 Oct 20, 2026
ade894a
refactor hardware drivers
wangxingjun778 Oct 22, 2026
6da5b04
bump version
wangxingjun778 Oct 22, 2026
cb8866b
refine hardware skills and support for new sdk for piper-x
wangxingjun778 Oct 23, 2026
9186fea
refactor and refine hardware for specific sys topology
wangxingjun778 Oct 24, 2026
8a8d51e
refactor & enhance hardware module and add multi-source for media sig…
wangxingjun778 Oct 26, 2026
42dcf8a
feat(vision): share camera capture and harden preview lifecycle
wangxingjun778 Oct 27, 2026
304d5fc
add new module: LeapCalib
wangxingjun778 Oct 27, 2026
30740ec
enhance LeapCalib
wangxingjun778 Oct 30, 2026
9191107
chore: normalize author identity via .mailmap
wangxingjun778 Nov 5, 2026
0bcc6ac
ok Merge branch 'feat/mhs' of github.com:modelscope/leapflow into fea…
wangxingjun778 Nov 5, 2026
e1c1170
chore(repo): ignore .qoder and normalize mislabeled author identity
wangxingjun778 Nov 5, 2026
d11b604
chore(repo): drop obsolete mailmap entry for the rewritten identity
wangxingjun778 Nov 5, 2026
8cd639c
refactor leapcalib
wangxingjun778 Nov 12, 2026
d20bde2
fix leapd arm halt issue
wangxingjun778 Nov 12, 2026
faf1c17
grasp box success
wangxingjun778 Nov 12, 2026
0916784
fix worker leakage
wangxingjun778 Nov 13, 2026
dbb8eef
add leapboard stop and manage
wangxingjun778 Nov 13, 2026
2c5420c
add plan-time reach pre-check to motion.sequence
wangxingjun778 Nov 13, 2026
36104c7
refine pipeline for NL control devices
wangxingjun778 Nov 13, 2026
86404e5
fix hang issue and real jetson board deployment
wangxingjun778 Nov 16, 2026
2669bb8
fix tui terminal crash
wangxingjun778 Nov 16, 2026
577426a
fix vision capabilities
wangxingjun778 Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 2 additions & 2 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
run: uv sync --all-extras --no-extra leapspace

- name: Lint
run: uv run ruff check src/leapflow/ src/benchmarks/ tests/benchmarks/ tests/ tools/
run: uv run ruff check src/leapflow/ src/leapcalib/ src/benchmarks/ tests/benchmarks/ tests/ tools/

- name: Verify derived fixtures match the cassette store
run: uv run python tools/sync_fixtures.py --check
Expand Down Expand Up @@ -111,7 +111,7 @@ jobs:
run: uv sync --all-extras --no-extra leapspace

- name: Lint
run: uv run ruff check src/leapflow/ src/benchmarks/ tests/benchmarks/ tests/ tools/
run: uv run ruff check src/leapflow/ src/leapcalib/ src/benchmarks/ tests/benchmarks/ tests/ tools/

- name: Verify derived fixtures match the stored exchanges
run: uv run python tools/sync_fixtures.py --check
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,8 @@ __marimo__/
.leapflow/
.learnings/
.idea/
# Qoder agent scratch space (artifacts, snapshots, vendored wheels)
.qoder/

# Swift / Xcode
.build/
Expand Down
7 changes: 7 additions & 0 deletions .mailmap
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Author identity normalization.
# Format: Proper Name <proper@email> Commit Name <commit@email>

# Same address, several display names.
班扬 <xingjun.wxj@alibaba-inc.com> Xingjun.Wang <xingjun.wxj@alibaba-inc.com>
班扬 <xingjun.wxj@alibaba-inc.com> xingjun.wxj <xingjun.wxj@alibaba-inc.com>
班扬 <xingjun.wxj@alibaba-inc.com> xingjun.wang <xingjun.wxj@alibaba-inc.com>
7 changes: 5 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,10 @@ The hardware stack (`leapflow/hardware/`, `leapflow/robot/`) is the agent's phys
- **Effects are typed; unknown is not success (MANDATORY)**: every write returns `WriteOutcome` with `side_effect_state` (`NONE`/`PARTIAL`/`UNKNOWN`/`COMMITTED`). An LHP error response means "could not attempt"; a `WriteOutcome` inside a success result means "might have landed". `ERR_TIMEOUT`/`ERR_CONNECTION_CLOSED`/`ERR_INTERNAL_ERROR` degrade to `UNKNOWN`, so an unproven command is verified before replay, never retried blindly — the same semantics the engine's `SideEffectState` gating consumes.
- **Control loops are node-local with a lease deadman**: high-frequency control (`control.v1`) runs on the node that owns the device; `control_sessions.py` owns lease open/renew/close, halts the device on lease expiry or disconnect, and records an `OperationVerdict` on close. `HardwareTools` routes `control_start` by transport kind and the node's advertised capability. The host never drives a `ControlBus` across the network — `HardwareRegistry.start_control_loop` refuses LHP-kind transports.
- **Physical signals are classified before they enter the pipeline**: hardware events carry `HardwareEventClass` (`REFLEX`/`DELIBERATIVE`/`RECORD`); raw readings never enter `SignalBuffer` or the causal graph — only derived, sampled events cross into the signal path, or device sampling rates flush every buffer. Readings are stamped on two clocks (`observed_at` wall time for audit correlation, `monotonic_at` for intervals/rates/staleness) — never derive staleness from wall-clock deltas. Device health and degradation are first-class signals: `DegradationCoordinator` executes declared failure modes fail-closed (`halt`, `hold_position`, `safe_return`, `switch_sensor`, `continue_blind`), and an incomplete degradation action falls back to immediate transport halt.
- **LeapRobot is a facade behind HCP, not a parallel hardware path (MANDATORY)**: `leapflow/robot/` (Robot protocol, motor bus, teleop, trajectory, VLA inference) is reachable only through the `robot_arm` and `vla_inference` transports; skills govern actuation through HCP channels plus approval/trust, never by importing `robot.Robot` directly. The Robot protocol has no halt member — `robot_arm` synthesizes halt as motion-to-zero-pose, which is not an emergency stop; an arm with a vendor-native stop needs its own `drivers/` module implementing `HardwareTransport.halt()` against the real stop.
- **LeapRobot is a facade behind HCP, not a parallel hardware path (MANDATORY)**: `leapflow/robot/` (Robot protocol, motor bus, teleop, trajectory, local inference strategies) is reachable only through the `robot_arm` transport, and remote policy/perception models only through the `model_service` transport; skills govern actuation through HCP channels plus approval/trust, never by importing `robot.Robot` directly. The Robot protocol has no halt member — `robot_arm` synthesizes halt as motion-to-zero-pose, which is not an emergency stop; an arm with a vendor-native stop needs its own `drivers/` module implementing `HardwareTransport.halt()` against the real stop.
- **Node skills extend the model's reach, never stand in for it (MANDATORY)**: node skills (`hardware/skills/`, discovered through `leapflow.hardware.skills`) give S2 capabilities to compose — metric motion, grounding, lifting pixels to 3D, grasp proposals, model-written motion programs (`motion.sequence`), VLA streaming — and own only what a model must never be trusted with: governed actuation, plan digests bound to run-scoped permits, freshness checks, typed side effects, and safety limits. No label matching, de-duplication, ambiguity refusal, scripted pick/place, or routing advice lives in a skill or its card: candidates and scores reach S2 verbatim, objects and grasps are referenced by the ids the node returned, and cards state capability, arguments, and limits (`test_builtin_cards_describe_capability_without_routing`). Built-in skills import only `skills.protocol`/`skills.ports`/`skills.geometry` and the stdlib. Every threshold comes from a declaration — device YAML, workcell, or Settings — validated at admission; a module constant that decides physical acceptance is a defect.
- **VLA output is embodied node-side, from declarations (MANDATORY)**: a model service runs the forward pass only and declares its native `action_space`/`observation_space`/`instruction_policy` in `model_info` (contract v1.1); the workcell's `policies.<name>` binds the model's groups to channels and cameras; `skills/adapt/` builds the pipeline (unnormalize → map → units → rotation → frame → reference → gripper → IK → validate, then aggregate/interpolate/smooth per tick) from those declarations alone — never a model-name special case. The SafetyShield's limits (envelope, rate, acceleration, first-action jump, contact) come from device declarations only; nothing the model or binding declares can widen them. `adapt/` imports only geometry, kinematics, and the stdlib (plus a lazily imported PIL for observation JPEG encoding), so `convert_to_lerobot.py` reuses its inverse and training encodes exactly what inference decodes. Every infer response echoes the model identity, and a mismatch with the admitted identity refuses the run.
- **Camera frames stay on the node unless two switches agree**: `perceive.snapshot` requires the workcell's `share_frames` *and* the host's `hardware.share_frames`; every shared frame is audited, enters the conversation as a volatile tool-result image (never the stable prefix, dropped first under compaction, described in text for providers without vision), and expires node-side. Everything else crosses LHP as geometry and typed outcomes only.
- **"LHP" names two layers — say which one you mean**: `lhp_gateway.py` is the LLM-Hardware Protocol semantic bridge (`hw_context_snapshot`), a layer above HCP that predates the wire-protocol naming; the wire protocol lives in `lhp/`. Name the layer when touching either, and do not move code between them to "unify" the acronym.

## Plugin and Extension Rules
Expand Down Expand Up @@ -191,7 +194,7 @@ LeapFlow exposes four levels of extension, ordered from lowest barrier to deepes

- **Path tree is a product contract**: every LeapFlow-managed path must be declared by `PathLayout`, `ProfileLayout`, `CacheLayout`, or a child layout object. Runtime code must consume layout APIs, never assemble managed paths with ad-hoc string joins.
- **Profile is the runtime boundary**: `profiles/<profile>/` owns profile metadata, config, DBs, memory, skills, gateway state, approval state, audit logs, runtime files, cache roots, and profile-scoped secrets. Cross-profile access requires an explicit layout object.
- **Workspace is context, not ownership**: workspace-local files are limited to `.leapflow/config.yaml` and `.leapflow/workspace.yaml`; profile data and caches stay under the active profile and are addressed by workspace/session ids.
- **Workspace is context, not ownership**: the only workspace-local file is the opt-in `.leapflow/config.yaml` override, created on demand by `leap config set --scope workspace` — running any command must not create anything in the user's directory. Workspace identity is derived (`workspace_id_for_path`), and the path-to-id record belongs to the profile's cache, not to a marker file in the workspace; profile data and caches stay under the active profile and are addressed by workspace/session ids.
- **Config is layered, not scattered**: durable settings live in `config/user.yaml`, `profiles/<profile>/config/*.yaml`, and optional workspace config. `LEAPFLOW_*` values are process overrides only; env files are not a supported configuration source.
- **`leap config` is the user-facing control plane**: every durable, user-writable setting must be discoverable and mutable through `ConfigService`, `leap config`, and TUI `/config`; do not add one-off setup commands, hidden YAML-only knobs, or new persistent env-first flows.
- **Config catalog is the discovery contract**: writable config fields must expose key, effective value, type, scopes, hot-reload semantics, category, value hint, and description. `leap config keys` stays compact and script-friendly; `leap config list` and `/config list` are the human-readable catalog; `leap config show <key>` and `/config show <key>` are the single-field detail views.
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ space-sync: ## Sync all dependencies including the leapspace extra
# `make lint` and CI can never disagree. leapspace is opt-in everywhere else and
# CI never syncs it (--no-extra leapspace), so it stays out of the lint gate too.
lint: ## Lint source code (mirrors the CI Lint step exactly)
uv run ruff check src/leapflow/ tests/ tools/
uv run ruff check src/leapflow/ src/leapcalib/ src/benchmarks/ tests/benchmarks/ tests/ tools/

# ── Test layers ───────────────────────────────────────────────────────────────
# The mock layer is broad and fast; the real layer is small, coarse, and never
Expand Down
19 changes: 11 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,20 +136,20 @@ LeapFlow bridges LLM intelligence and real-world hardware through a **three-laye
HCP per-device contract
──────────────── 6-method transport: open / close / read / write / probe / halt
Southbound transport plugins
──────────────── ROS2 · Feetech · RealSense · VLA · MCP · media · simulated …
──────────────── ROS2 · Piper · Feetech · RealSense · model service · MCP · media …
```

```
System-2 (LLM) 0.2–1 Hz task planning → SubtaskGoal
System-2 (LLM) 0.2–1 Hz task planning → node skill (plan → approve → run)
↓
System-1.5 (VLA) 5–30 Hz observation + goal → action chunks
System-1.5 (skills) 5–30 Hz perceive · grasp · policy.run (VLA action chunks)
↓
System-1 (servo) 100–1000 Hz PID / impedance — never crosses the fabric
↓
actuators escalation flows back up ↑
```

The host sends **goals**; the node decomposes them locally. A 1 kHz servo loop stays safe regardless of network latency.
The host sends **goals**; the node decomposes them locally. A skill is planned on the node, approved once by its plan digest, then executed next to the device with a run-scoped permit — so a 1 kHz servo loop and a 30 Hz policy stream stay safe regardless of network latency.

**Protocol once, devices for free** — new device = one YAML on the node; new protocol = one transport plugin; new node = one fleet entry.

Expand All @@ -163,7 +163,8 @@ The host sends **goals**; the node decomposes them locally. A 1 kHz servo loop s
| Safety | `hw_estop` | no approval — node-local |
| Control | `hw_control_start` · `hw_control_stop` · `hw_control_status` | leased deadman |
| Fleet | `hw_fleet_status` · `hw_fleet_discover` · `hw_fleet_halt_all` | multi-node |
| Policy | `hw_policy_infer` · `hw_policy_episode` · `hw_policy_list` | VLA inference |
| Skills | `hw_skill_list` · `hw_skill_run` · `hw_skill_status` · `hw_skill_stop` | plan-digest approval, node-local |
| Data | `hw_skill_label` | labels a recorded episode's outcome |

</details>

Expand All @@ -180,8 +181,9 @@ The host sends **goals**; the node decomposes them locally. A 1 kHz servo loop s
<details>
<summary>Bundled physical-AI stack</summary>

- **Robot arms** (`leapflow.robot`) — LeRobot-derived: serial bus (Feetech), teleop, camera capture, episode export, policy registry (`vla_local` / `vla_remote` / `position_control`).
- **VLA as a device** — policy server (gRPC or HTTP) appears as an HCP virtual device; inference is a CONFIGURE effect — physical motion happens only when the returned action is written to an actuator.
- **Robot arms** (`leapflow.robot`) — LeRobot-derived: serial bus (Feetech), teleop, camera capture, episode export, policy registry (`vla_local` / `position_control`).
- **Node skills** — calibration (`calib.*`), perception (`perceive.locate`), grasp proposals (`grasp.propose`), model-written motion programs (`motion.sequence`), and learned policies (`policy.run`) run on the node behind the same gate chain; with an episodes directory configured, every actuating run is recorded as a training episode.
- **Models as devices** — a policy or perception server (`model_service` transport, HTTP) appears as an HCP device; inference never moves anything — only a skill's gated motion writes to actuators.
- **Depth cameras** — RealSense integration; frames as governed binary channels.
- **ROS2** — sidecar transport over Unix socket; topics/services/actions map into HCP.

Expand Down Expand Up @@ -317,7 +319,7 @@ Inside the TUI: `/config list`, `/config set …` — hot-reloads the active ses
|-----|---------|-------------|
| `llm.model` | — | Primary model |
| `llm.base_url` | — | OpenAI-compatible endpoint |
| `llm.context_length` | `1000000` | Context budget |
| `llm.context_length` | `1000000` | Runtime context budget shown in the TUI status bar |
| `memory.working_max_tokens` | `12000` | Working memory budget |
| `tools.terminal_session_enabled` | `false` | Persistent terminal sessions |
| `dashboard.enabled` | `true` | LeapBoard |
Expand Down Expand Up @@ -412,6 +414,7 @@ Drop a YAML manifest in `~/.leapflow/profiles/<profile>/gateway/manifests/`, imp
| `leap skills list\|show\|export\|import` | Skill library |
| `leap evolve` | Run the learning boundary |
| `leap board` | LeapBoard dashboard |
| `leap board status\|stop [--all]` | List / stop LeapBoard web servers (incl. orphans) |
| `leap hw list\|serve\|verify\|estop\|…` | Hardware / LHP node |
| `leap daemon start\|stop\|restart\|status` | Manage leapd |
| `leap config list\|set\|show` | Configuration |
Expand Down
Loading
Loading