Skip to content

Feature & Refactor: Node Skills, LeapCalib & Shared Vision - #56

Merged
wangxingjun778 merged 29 commits into
mainfrom
feat/mhs
Nov 16, 2026
Merged

wangxingjun778 merged 29 commits into
mainfrom
feat/mhs

Conversation

@wangxingjun778

Copy link
Copy Markdown
Member

Features

Node skill runtime (S1.5)

  • New skills runtime: discover → plan (read-only, digest-bound) → human approve-by-digest → execute under a run-scoped permit → stop. Device access only through SkillContext ports, so swapping arm/camera/model never touches a skill.
  • skill.v1 LHP methods plus host tools hw_skill_list/run/status/stop/label with wire-plan verification (skill_tools.py).
  • Built-in skill packs registered via entry points: motion, motion.sequence, grasp.propose/cycle/verify, perceive, scene/frame, calib.* (camera, governed, bridge, metric readiness), policy.run, episode.
  • Episode recording for every actuating run (arm + camera samples, governed batches, policy chunks), numpy-free replayable layout.
  • New node write gate: one governed chain (writable → envelope → interlocks → degradation → rate → safety → I/O lock) shared by host writes and node-local skill motion.

LeapCalib

  • New standalone leapcalib package: typed SI contracts, SE3 geometry, estimation (metric motion, task/raw response, hand-eye, registration), validation bundles, and leap-calib CLI; [leapcalib] extra, numpy only during solves.
  • leapflow.hardware.calibration boundary with conversion adapter, frame lifecycle (promotion/attestation), bridge assessment/policy, provenance, rollback, readiness, generation tracking.

Vision & perception

  • Shared frame hub: one acquisition pump per camera, many consumers, latest-frame semantics, derived views computed outside lifecycle locks.
  • CaptureGroup soft-synchronised multi-camera capture with spread measurement and bounded re-capture.
  • Node-local RGB-D: FrameSource, aligned depth + intrinsics, quality/geometry/observation modules.
  • LHP read_frame gains min_sequence / max_age_ms; capture_rgbd retains raw depth.
  • Scene continuity: scene memory, fusion, mosaic scene view, pixel-space change signatures, per-view table plane.
  • LLM vision: evidence-based vision capability probe, area-based image token accounting, and tool images delivered as volatile user-message attachments.

Models as devices

  • model_service transport + v1.1 model-serving contract (grounding, policy, caption), fail-closed identity/checksum verification on open and refresh.
  • policy.run streams action chunks through the workcell binding and safety shield; replaces vla_inference / vla_remote.

Hardware services

  • Advisory limits: monitor-only thresholds (temperature, voltage, firmware flags) reported with exponential backoff, never re-gated.
  • Piper-X driver on the official pyAgxArm SDK (model/firmware-aware limits and codec).
  • LeapBoard lifecycle: leap board status|stop [--all], including orphaned server discovery with proven identity before signalling.
  • Reference workcell under tools/piper: ops CLI, deploy unit, serve/restart/estop scripts.

Refactor

  • Hardware module re-cut by concern — drivers/, transports/, lhp/, skills/, vision/, calibration/, observability/ — with enforced dependency direction.
  • Removed superseded paths: capability_router, control_hierarchy, physical_skill tool plugin, vla_inference, vla_remote.
  • Engine: new result budget (meaning-preserving tool-result shrinking), tool attachment handling moved out of message helpers.
  • LHP: authority, param codec, permit digest binding, control-session and server split.
  • Daemon: turn/stream ownership separated from bookkeeping; named cancel scoped to its own turn.
  • CLI leap hw verify/validate reworked around attestation and fleet-node device binding.

Enhance

  • Reachability pre-check at plan time for motion.sequence (refuses unreachable poses before approval).
  • Degradation: sensor-loss streak tracking, health monitor, observability digest/exporter/inventory.
  • Cache manager: per-path locking, schema-once init, legacy index removal, refresh-on-reinsert.
  • TUI: wide-glyph-safe render guard, persistent input history, status/context plumbing.
  • Config: advisory limits, USB video exclusion, preview FPS/size/idle ceilings, skill-runtime and episode settings.
  • 81 new test modules covering skills, calibration, vision, LHP, daemon and TUI paths.

Fix

  • leapd hang/leak: stream teardown now awaits cancelled children, closes with aclosing, releases the session lock before bookkeeping, tolerates foreign-Context finalizers, and reports session-busy instead of a blank wait.
  • TUI crash on macOS Terminal: double-width glyph repaint could SIGBUS the terminal; repainting bypassed via render guard.
  • Worker/process leakage (calibration worker, cache handles, dashboard servers, reading store).
  • Arm halt on transient sensor loss incorrectly latched; corrected streak semantics.
  • False "model cannot view images" verdicts; image budget miscounts.
  • LeapBoard orphaned servers holding ports after launcher failure.

Docs

  • README: three-layer stack, hw_skill_* tool surface, models-as-devices, leap board status|stop.
  • docs/plugins/hardware_peripherals_board.md: new §9 on shared vision, FrameHub, capture groups and ROS distribution; updated media/preview limits.
  • AGENTS.md: node-skill and gate invariants.

Upgrade notes

  • piper_sdk is no longer used; the Piper transport requires pyAgxArm on the CAN-owning node.
  • vla_remote policy backend and hw_policy_* tools are gone — use model_service + policy.run.
  • New extras: leapcalib, vision; new console script leap-calib; new entry-point groups leapcalib.methods and leapflow.hardware.skills.
  • tools/piper/data/runs currently ships committed node run artifacts (~176 files); consider ignoring it in the repo.

wangxingjun778 and others added 29 commits October 13, 2026 16:50
Add grouped freshness, typed health signals, optional ROS previews and native-depth encoders. Drain daemon clients during graceful shutdown and cover ownership, release, and restart paths with regression tests.
Signed-off-by: 班扬 <xingjun.wxj@alibaba-inc.com>
- Stop tracking .qoder/ (Qoder agent scratch space: build artifacts, repo snapshots and vendored wheels, ~70MB) and ignore it going forward. Local files are retained.
- Map the tooling-mislabeled "Cheney Zhang <chen.zhang@zilliz.com>" author, written by a commit skill that hardcoded --author, to the committer's real identity via .mailmap. Published history on main and the release branches is deliberately not rewritten.

Signed-off-by: wangxingjun778 <wangxingjun778@163.com>
The 18 commits whose author field was overridden by the commit skill are now rewritten in history, so the display-layer alias is dead weight.

Signed-off-by: wangxingjun778 <wangxingjun778@163.com>
@wangxingjun778
wangxingjun778 merged commit 4910ae6 into main Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant