Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
c3a2a1a
merge
wangxingjun778 Oct 13, 2026
950b941
ok Merge branch 'feat/mhs' of github.com:modelscope/leapflow into fea…
wangxingjun778 Oct 15, 2026
acd9464
ok Merge branch 'feat/mhs' of github.com:modelscope/leapflow into fea…
wangxingjun778 Oct 15, 2026
78a2563
remove unused content
wangxingjun778 Oct 16, 2026
6a5aea3
refactor and enhance hardware framework
wangxingjun778 Oct 19, 2026
e5b133d
add enhance hardware skills
wangxingjun778 Oct 20, 2026
1093d18
ok Merge branch 'feat/mhs' of github.com:modelscope/leapflow into fea…
wangxingjun778 Oct 20, 2026
ade894a
refactor hardware drivers
wangxingjun778 Oct 22, 2026
6da5b04
bump version
wangxingjun778 Oct 22, 2026
cb8866b
refine hardware skills and support for new sdk for piper-x
wangxingjun778 Oct 23, 2026
9186fea
refactor and refine hardware for specific sys topology
wangxingjun778 Oct 24, 2026
8a8d51e
refactor & enhance hardware module and add multi-source for media sig…
wangxingjun778 Oct 26, 2026
42dcf8a
feat(vision): share camera capture and harden preview lifecycle
wangxingjun778 Oct 27, 2026
304d5fc
add new module: LeapCalib
wangxingjun778 Oct 27, 2026
30740ec
enhance LeapCalib
wangxingjun778 Oct 30, 2026
9191107
chore: normalize author identity via .mailmap
wangxingjun778 Nov 5, 2026
0bcc6ac
ok Merge branch 'feat/mhs' of github.com:modelscope/leapflow into fea…
wangxingjun778 Nov 5, 2026
e1c1170
chore(repo): ignore .qoder and normalize mislabeled author identity
wangxingjun778 Nov 5, 2026
d11b604
chore(repo): drop obsolete mailmap entry for the rewritten identity
wangxingjun778 Nov 5, 2026
8cd639c
refactor leapcalib
wangxingjun778 Nov 12, 2026
d20bde2
fix leapd arm halt issue
wangxingjun778 Nov 12, 2026
faf1c17
grasp box success
wangxingjun778 Nov 12, 2026
0916784
fix worker leakage
wangxingjun778 Nov 13, 2026
dbb8eef
add leapboard stop and manage
wangxingjun778 Nov 13, 2026
2c5420c
add plan-time reach pre-check to motion.sequence
wangxingjun778 Nov 13, 2026
36104c7
refine pipeline for NL control devices
wangxingjun778 Nov 13, 2026
86404e5
fix hang issue and real jetson board deployment
wangxingjun778 Nov 16, 2026
2669bb8
fix tui terminal crash
wangxingjun778 Nov 16, 2026
577426a
fix vision capabilities
wangxingjun778 Nov 16, 2026
32281f8
ok Merge branch 'main' of github.com:modelscope/leapflow into feat/mhs
wangxingjun778 Nov 16, 2026
524841a
bump version
wangxingjun778 Nov 16, 2026
6f87152
fix hw_skill_run args issue
wangxingjun778 Nov 16, 2026
5eaffb1
fix piper-x halt policy and add hold-in-place policy
wangxingjun778 Nov 17, 2026
5857393
fix tui crash; enhance grasp async-batch perception
wangxingjun778 Nov 17, 2026
55c4408
fix config health classification and guard live preview lifecycle
wangxingjun778 Nov 17, 2026
ebef553
fix candidates and leapboard monitor
wangxingjun778 Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,9 +117,10 @@ The hardware stack (`leapflow/hardware/`, `leapflow/robot/`) is the agent's phys
- **Vendor drivers live under `drivers/`, generic mechanisms under `transports/` (MANDATORY)**: a single-vendor SDK integration (e.g. `drivers/realsense.py`) registers through the same `leapflow.hardware.transports` entry-point namespace; a device-named module under `transports/` fails `test_hardware_transports_are_not_named_after_one_device`. Vendor SDK imports must be lazy — inside the node-local `open()` path — and the SDK must never enter project dependencies, because the host process never installs it. Capture stays on the node; only bounded previews and calibration metadata cross into `Reading`/LHP frames.
- **Writes pass the full gate chain in order, and enforcement re-runs where execution happens**: envelope bounds → interlock evaluation → `check_safety_policy` (re-run by `LhpNodeServer` before every write) → approval feasibility (`WriteGovernor`) → trust gate (progressive `UNTRUSTED → CANDIDATE → VERIFIED → PRODUCTION`; only reversible channels at VERIFIED+ may skip approval, irreversible channels always prompt) → for remote `permit_required` nodes, an Ed25519 `ExecutionPermit` (action digest, single-use nonce, audience binding, TTL) issued by the host and verified node-side. Enforcement deliberately lives on the node so bypassing the host transport cannot skip it. Trust is promoted by verified settled outcomes (`OperationVerdict`), never by transport write success alone.
- **Effects are typed; unknown is not success (MANDATORY)**: every write returns `WriteOutcome` with `side_effect_state` (`NONE`/`PARTIAL`/`UNKNOWN`/`COMMITTED`). An LHP error response means "could not attempt"; a `WriteOutcome` inside a success result means "might have landed". `ERR_TIMEOUT`/`ERR_CONNECTION_CLOSED`/`ERR_INTERNAL_ERROR` degrade to `UNKNOWN`, so an unproven command is verified before replay, never retried blindly — the same semantics the engine's `SideEffectState` gating consumes.
- **A halt holds in place; it never releases torque (MANDATORY)**: `halt()` is a controlled stop — actuators are commanded to the position they report *now* (or a declared output-off for streams), never to a zero vector, a home pose, or a vendor emergency stop that cuts power, because a dropped arm with a payload is the failure the stop exists to prevent. Every `TransportStatus` from `halt()`/`probe()` reports what the stop physically did as `stop_effect` (`hold`/`output_off`/`device_latched`/`none`; `""` means unsaid, never assumed) and whether it was observed as `stop_verified` (`True`/`False`/`None`); LHP carries both verbatim (`TransportStatus.stop_fields`), `hw_estop` audits them and warns to use the physical stop when unverified, and a skill run records them as evidence. A transport that cannot derive a hold target from fresh state sends nothing and says so — a partial vector would invent targets. ROS2 command topics declare their stop in `halt_commands` (`zero` or `{hold_from: <state channel>}`); an undeclared topic is never published to and halt is advertised only when every command topic is declared. Torque release (reset/disable/latch resume) happens only through a governed enable under the driver's declared `TorqueReleasePolicy` (`refuse` or `at_rest`), never as a side effect of stopping.
- **Control loops are node-local with a lease deadman**: high-frequency control (`control.v1`) runs on the node that owns the device; `control_sessions.py` owns lease open/renew/close, halts the device on lease expiry or disconnect, and records an `OperationVerdict` on close. `HardwareTools` routes `control_start` by transport kind and the node's advertised capability. The host never drives a `ControlBus` across the network — `HardwareRegistry.start_control_loop` refuses LHP-kind transports.
- **Physical signals are classified before they enter the pipeline**: hardware events carry `HardwareEventClass` (`REFLEX`/`DELIBERATIVE`/`RECORD`); raw readings never enter `SignalBuffer` or the causal graph — only derived, sampled events cross into the signal path, or device sampling rates flush every buffer. Readings are stamped on two clocks (`observed_at` wall time for audit correlation, `monotonic_at` for intervals/rates/staleness) — never derive staleness from wall-clock deltas. Device health and degradation are first-class signals: `DegradationCoordinator` executes declared failure modes fail-closed (`halt`, `hold_position`, `safe_return`, `switch_sensor`, `continue_blind`), and an incomplete degradation action falls back to immediate transport halt.
- **LeapRobot is a facade behind HCP, not a parallel hardware path (MANDATORY)**: `leapflow/robot/` (Robot protocol, motor bus, teleop, trajectory, local inference strategies) is reachable only through the `robot_arm` transport, and remote policy/perception models only through the `model_service` transport; skills govern actuation through HCP channels plus approval/trust, never by importing `robot.Robot` directly. The Robot protocol has no halt member — `robot_arm` synthesizes halt as motion-to-zero-pose, which is not an emergency stop; an arm with a vendor-native stop needs its own `drivers/` module implementing `HardwareTransport.halt()` against the real stop.
- **LeapRobot is a facade behind HCP, not a parallel hardware path (MANDATORY)**: `leapflow/robot/` (Robot protocol, motor bus, teleop, trajectory, local inference strategies) is reachable only through the `robot_arm` transport, and remote policy/perception models only through the `model_service` transport; skills govern actuation through HCP channels plus approval/trust, never by importing `robot.Robot` directly. The Robot protocol has no halt member — `robot_arm` synthesizes halt by sending a fresh observation back as the target (hold in place), which is not an emergency stop; an arm with a vendor-native controlled stop needs its own `drivers/` module implementing `HardwareTransport.halt()` under the hold-in-place contract.
- **Node skills extend the model's reach, never stand in for it (MANDATORY)**: node skills (`hardware/skills/`, discovered through `leapflow.hardware.skills`) give S2 capabilities to compose — metric motion, grounding, lifting pixels to 3D, grasp proposals, model-written motion programs (`motion.sequence`), VLA streaming — and own only what a model must never be trusted with: governed actuation, plan digests bound to run-scoped permits, freshness checks, typed side effects, and safety limits. No label matching, de-duplication, ambiguity refusal, scripted pick/place, or routing advice lives in a skill or its card: candidates and scores reach S2 verbatim, objects and grasps are referenced by the ids the node returned, and cards state capability, arguments, and limits (`test_builtin_cards_describe_capability_without_routing`). Built-in skills import only `skills.protocol`/`skills.ports`/`skills.geometry` and the stdlib. Every threshold comes from a declaration — device YAML, workcell, or Settings — validated at admission; a module constant that decides physical acceptance is a defect.
- **VLA output is embodied node-side, from declarations (MANDATORY)**: a model service runs the forward pass only and declares its native `action_space`/`observation_space`/`instruction_policy` in `model_info` (contract v1.1); the workcell's `policies.<name>` binds the model's groups to channels and cameras; `skills/adapt/` builds the pipeline (unnormalize → map → units → rotation → frame → reference → gripper → IK → validate, then aggregate/interpolate/smooth per tick) from those declarations alone — never a model-name special case. The SafetyShield's limits (envelope, rate, acceleration, first-action jump, contact) come from device declarations only; nothing the model or binding declares can widen them. `adapt/` imports only geometry, kinematics, and the stdlib (plus a lazily imported PIL for observation JPEG encoding), so `convert_to_lerobot.py` reuses its inverse and training encodes exactly what inference decodes. Every infer response echoes the model identity, and a mismatch with the admitted identity refuses the run.
- **Camera frames stay on the node unless two switches agree**: `perceive.snapshot` requires the workcell's `share_frames` *and* the host's `hardware.share_frames`; every shared frame is audited, enters the conversation as a volatile tool-result image (never the stable prefix, dropped first under compaction, described in text for providers without vision), and expires node-side. Everything else crosses LHP as geometry and typed outcomes only.
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ The host sends **goals**; the node decomposes them locally. A skill is planned o
|---|---|---|
| Inspect | `hw_list` · `hw_describe` · `hw_read` · `hw_status` | read-only |
| Act | `hw_configure` · `hw_actuate` · `hw_dispense` · `hw_batch_actuate` | approval-gated |
| Safety | `hw_estop` | no approval — node-local |
| Safety | `hw_estop` | no approval — node-local; holds in place, reports `stop_effect` / `stop_verified` |
| Control | `hw_control_start` · `hw_control_stop` · `hw_control_status` | leased deadman |
| Fleet | `hw_fleet_status` · `hw_fleet_discover` · `hw_fleet_halt_all` | multi-node |
| Skills | `hw_skill_list` · `hw_skill_run` · `hw_skill_status` · `hw_skill_stop` | plan-digest approval, node-local |
Expand Down Expand Up @@ -205,7 +205,8 @@ The host sends **goals**; the node decomposes them locally. A skill is planned o
pip install leapflow
openssl rand -hex 24 > ~/.leapflow/.lhp-token && chmod 600 ~/.leapflow/.lhp-token
LEAPFLOW_LHP_TOKEN="$(cat ~/.leapflow/.lhp-token)" \
leap hw serve --node-id edge01 --host 0.0.0.0 --port 8767
leap hw serve --node-id edge01 --host 0.0.0.0 --port 8767 \
--log-file ~/.leapflow/logs/lhp-node.log # size-rotated; config hardware.node_log_*

# Host
export EDGE_LHP_TOKEN='paste-the-token-here'
Expand Down
10 changes: 10 additions & 0 deletions docs/plugins/third_party_plugin_development.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,16 @@ config, gateway dispatch) plus the Tool Capability Contract in

Additionally, `FrameStore` (`perception/storage/frame_store.py`) is a `@runtime_checkable` Protocol for pluggable frame persistence backends.

Node skills may implement `PreparedPlanningSkill` (`hardware/skills/planning.py`) to
prepare immutable jobs for bounded parallel evaluation. Two optional, runtime-checkable
extensions keep domain decisions in the plugin: `PlanningAlternativesSkill.alternatives`
offers bounded task-equivalent programs after an infeasible single request;
`PlanConfirmationSkill.confirm` performs read-only live checks before a selected plan
is sealed. `PlanningSettings.auto_variants` defaults to true; no alternative executes
without the normal approval, permit, freshness, and lease gates. `ScenePort.grasps`
returns live sibling proposals, while `relocate(commit=False)` measures without
refreshing object, grasp, or observation memory.

**When to use each:**

- **ToolPlugin** — You want the LLM agent to invoke your functionality as a tool call (most common).
Expand Down
10 changes: 7 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -88,14 +88,14 @@ leapspace = [
# motion/response solves; no SciPy/OpenCV dependency is needed by these methods.
leapcalib = ["numpy>=1.24"]
# Lightweight opt-in preview encoder; no policy-learning or ROS dependencies.
vision = ["numpy>=1.24", "opencv-python>=4.8"]
vision = ["numpy>=1.24", "opencv-python>=4.8,<5"]
robot = [
"torch>=2.0",
"torchvision>=0.15",
"safetensors>=0.4",
"huggingface-hub>=0.20",
"numpy>=1.24",
"opencv-python>=4.8",
"opencv-python>=4.8,<5",
"pyserial>=3.5",
"Pillow>=10.0",
]
Expand Down Expand Up @@ -143,10 +143,14 @@ builtin_frame_lifecycle = "leapflow.hardware.skills.builtin.frame_lifecycle:skil
builtin_grasp_verify = "leapflow.hardware.skills.builtin.grasp_verify:skills"
builtin_grasp_cycle = "leapflow.hardware.skills.builtin.grasp_cycle:skills"
builtin_scene = "leapflow.hardware.skills.builtin.scene:skills"
builtin_survey = "leapflow.hardware.skills.builtin.survey:skills"
builtin_sequence = "leapflow.hardware.skills.builtin.sequence:skills"
builtin_policy = "leapflow.hardware.skills.builtin.policy:skills"
builtin_episode = "leapflow.hardware.skills.builtin.episode:skills"

[project.entry-points."leapflow.hardware.view_planners"]
ring = "leapflow.hardware.skills.view_planner:RingViewPlanner"

[build-system]
requires = ["setuptools>=68.0"]
build-backend = "setuptools.build_meta"
Expand All @@ -171,7 +175,7 @@ include = ["leapflow*", "leapspace*", "leapcalib*", "benchmarks*"]
# non-identifier `task-001`). Auto-discovery already ships each task's
# action.py; this keeps its config.yaml beside it so an installed example task
# stays runnable rather than half-packaged.
"leapspace.app_space.tasks.task-001" = ["*.yaml"]
"leapspace.app_space" = ["tasks/*/*.yaml"]

[tool.pytest.ini_options]
asyncio_mode = "auto"
Expand Down
6 changes: 5 additions & 1 deletion src/leapflow/cli/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -332,7 +332,7 @@ def main(argv: list[str] | None = None) -> int:
hw_read.add_argument("channel", help="Channel id from `leap hw describe`")
hw_status = hw_sub.add_parser("status", parents=[hw_json], help="Show transport health and recent events")
hw_status.add_argument("device", nargs="?", default="", help="Optional device id; omit to roll up every device")
hw_estop = hw_sub.add_parser("estop", parents=[hw_json], help="Emergency-stop a device (no approval required)")
hw_estop = hw_sub.add_parser("estop", parents=[hw_json], help="Stop a device in place, holding actuators under power (no approval required)")
hw_estop.add_argument("device", help="Device id from `leap hw list`")
hw_pause = hw_sub.add_parser("pause", parents=[hw_json], help="Pause daemon sampling for a device")
hw_pause.add_argument("device", help="Device id from `leap hw list`")
Expand All @@ -349,6 +349,10 @@ def main(argv: list[str] | None = None) -> int:
hw_serve.add_argument("--port", type=int, default=8767, help="TCP bind port (default: 8767; 0 picks a free port)")
hw_serve.add_argument("--unix", default="", dest="unix_path", help="Bind a Unix socket path instead of TCP")
hw_serve.add_argument("--token", default="", help="Shared hello secret (env LEAPFLOW_LHP_TOKEN)")
hw_serve.add_argument(
"--log-file", default="", help="Append and size-rotate the node log here (config hardware.node_log_file)"
)
hw_serve.add_argument("--verbose", action="store_true", help="Log at DEBUG level")
# Verification subcommands
hw_verify = hw_sub.add_parser("verify", parents=[hw_json], help="Verify a device declaration (digest-bound)")
hw_verify.add_argument("device", help="Device id to verify")
Expand Down
34 changes: 27 additions & 7 deletions src/leapflow/cli/commands/hardware.py
Original file line number Diff line number Diff line change
Expand Up @@ -163,13 +163,29 @@ async def _run_serve(args: argparse.Namespace) -> int:
"(--node-id, or LEAPFLOW_LHP_NODE_ID)"
)
return 1
return await serve_node(
node_id=node_id,
host=str(getattr(args, "host", "") or "127.0.0.1"),
port=int(getattr(args, "port", 0)),
unix_path=str(getattr(args, "unix_path", "") or ""),
token=token,
from leapflow.logging_setup import init_node_logging

log_path = init_node_logging(
load_config(),
log_file=str(getattr(args, "log_file", "") or ""),
verbose=bool(getattr(args, "verbose", False)),
)
if log_path is not None:
# stderr, not stdout: stdout carries the machine-readable ready line.
sys.stderr.write(f"node log: {log_path}\n")
try:
return await serve_node(
node_id=node_id,
host=str(getattr(args, "host", "") or "127.0.0.1"),
port=int(getattr(args, "port", 0)),
unix_path=str(getattr(args, "unix_path", "") or ""),
token=token,
)
except Exception:
# The node's own log is where a field failure is read afterwards, so the
# reason it died belongs there, not only on a console nobody kept.
logger.critical("LHP node %r exited on an unhandled error", node_id, exc_info=True)
raise


# ── Read / estop plane ─────────────────────────────────────────────────────
Expand Down Expand Up @@ -398,7 +414,11 @@ def _render_one_status(report: dict[str, Any]) -> None:


def _render_estop(result: dict[str, Any]) -> None:
_ok(f"{result.get('device_id')}: halted={result.get('halted')}")
effect = result.get("stop_effect") or "unreported"
_ok(f"{result.get('device_id')}: halted={result.get('halted')} effect={effect} "
f"verified={result.get('stop_verified')}")
if result.get("warning"):
_warn(str(result["warning"]))


def _render_sampling(result: dict[str, Any]) -> None:
Expand Down
2 changes: 2 additions & 0 deletions src/leapflow/cli/commands/hardware_workcells.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ def _render(result: dict[str, Any]) -> None:
workcells = result["workcells"]
for workcell_id in workcells["admitted"]:
print(f" {_GREEN}\u2713{_RESET} {workcell_id}: admitted")
for note in workcells.get("advisories") or ():
print(f" {_YELLOW}!{_RESET} {note['workcell']}: {note['detail']}")
for note in workcells["rejected"]:
name = note["workcell"] or Path(note["source"]).name
print(f" {_RED}\u2717{_RESET} {name}: {note['detail']}")
Expand Down
16 changes: 16 additions & 0 deletions src/leapflow/cli/commands/slash_handlers.py
Original file line number Diff line number Diff line change
Expand Up @@ -1782,6 +1782,22 @@ def render_plugin_payload(console: "LeapConsole", payload: dict[str, Any]) -> No
info.append(f"Tools ({len(tools)}):")
for t in tools:
info.append(f"\n - {t.get('name')}: {t.get('description', '')[:50]}")
usage = payload.get("usage") or []
if usage:
# Real failures, feasibility refusals and self-repairs are read apart:
# only the first says the tool is broken.
info.append("\nUsage (fail / refused / repaired):")
for row in usage:
codes = ", ".join(
f"{code}×{count}" for code, count in (row.get("failure_codes") or {}).items()
)
info.append(
f"\n - {row.get('tool')}: {row.get('calls', 0)} calls, "
f"{float(row.get('failure_rate') or 0):.0%} / "
f"{float(row.get('refusal_rate') or 0):.0%} / "
f"{float(row.get('repair_rate') or 0):.0%}"
+ (f" — {codes}" if codes else "")
)
console.print(Panel(info, title=str(payload.get("plugin_id") or "Plugin"), border_style="cyan"))
return

Expand Down
Loading
Loading