From d8c27917e889c2e2020fd703e5b9188bd57905d8 Mon Sep 17 00:00:00 2001 From: Jack Champagne Date: Mon, 12 Oct 2026 11:30:55 -0400 Subject: [PATCH] Update peer fleet setup guidance --- .../skills/add-fleet-device/SKILL.md | 59 ++++++++-------- .../extension/skills/create-a-fleet/SKILL.md | 70 +++++++++---------- .../skills/troubleshoot-fleet/SKILL.md | 52 ++++++++------ tools/fleet/README.md | 35 +++++----- 4 files changed, 111 insertions(+), 105 deletions(-) diff --git a/packages/extension/skills/add-fleet-device/SKILL.md b/packages/extension/skills/add-fleet-device/SKILL.md index 23bd6d41b..f08d1a831 100644 --- a/packages/extension/skills/add-fleet-device/SKILL.md +++ b/packages/extension/skills/add-fleet-device/SKILL.md @@ -27,7 +27,7 @@ what you say: | hub / server / canonical | "home base" — the machine that hosts your work | | enroll / join token | "link this machine in" | | client | "a window onto home base" | -| standalone + advertise `serving` (peer) | "its own studio you can hop to" | +| standalone + advertise `serving` (peer) | "its own studio, with fleet sessions and prompts syncing between machines" | | verify-attach / roster health | "check the link actually works" | | transport (ssh / tailscale) | "how the machines reach each other" | @@ -57,18 +57,15 @@ first"). Do not try to stand up home base here. Frame the topology choice as an outcome, never with the words "hub" or "peer": -> **"When this machine is away from home base (or home base is off), what should happen?"** -> - **"It's my window into home base — I mostly work off the other machine"** → a -> window onto home base (recommended, and the fully-wired path: your sessions -> follow you, this machine shows the same work). -> - **"It should fully work on its own, and I'll hop between the two"** → its own -> studio. +> **"Should this machine run its own studio, or depend on one host?"** +> - **"Run its own studio, and keep fleet sessions in sync"** (recommended) → peer +> studio. Each machine has its own engine; session activity and prompts sync +> between them in near real time. +> - **"Use one host for all my work"** → a window onto home base. The other machine +> depends on that host being reachable. -If they choose "its own studio," add the honest caveat immediately: *"You can set -that up today and hop between them — but running a live chat directly on the other -machine's engine is still landing, so the smoothest 'both machines as one' is -keeping one as home base. Want the window setup instead, or the independent one -with that caveat?"* Then honor their choice. +Honor their choice. Do not describe peer studios as simply hopping between isolated +histories: session activity and prompts sync between peer machines. ## Step 3 — pick the one machine @@ -80,10 +77,15 @@ you like"). ## Step 4 — readiness: can the two machines already reach each other? -Ask the plain question first, before anything is changed: +First choose the connection method in Step 5. SSH is not a prerequisite when the +machines will connect over Tailscale. For Tailscale, check both devices are signed +in and online, and that MagicDNS resolves. If it is not set up, direct the user to +[Tailscale's install guide](https://tailscale.com/kb/1017/install) and resume when +they are ready. Do not ask for router port-forwarding. -> **"Have you ever connected to this machine over SSH from here? (If you've `ssh`'d -> into it before, yes.)"** +For SSH, ask the plain question first, before anything is changed: + +> **"Have you ever connected to this machine over SSH from here? (If you've `ssh`'d into it before, yes.)"** Then **probe silently** and only surface a prerequisite if it actually fails: @@ -101,20 +103,20 @@ pipeline: I'll retry." (Behind the scenes: enable the service, append this machine's public key to the target's `~/.ssh/authorized_keys`, add a `Host ` block to `~/.ssh/config`.) -- **No SSH alias** → the enroll step requires one; add the `Host ` block - before proceeding (the link install refuses without it). +- **No SSH alias** → add the `Host ` block only if SSH is the chosen + connection method. ## Step 5 — how the machines connect (transport), by OS -Default is **SSH** — the universal floor, works everywhere. Choose **Tailscale** -only when the machine roams between networks *and* Tailscale is actually up: +Recommend **Tailscale** when the machines may be on different or changing networks. +It provides private reachability without router port-forwarding. If it is not +installed or signed in, direct the user to [Tailscale's install guide](https://tailscale.com/kb/1017/install) +and pause until setup is complete. Use **SSH** for a trusted reachable network or +when the user prefers SSH: -- Before choosing Tailscale, verify both machines are online in `tailscale status`, - MagicDNS resolves, and home base is sharing its engine with `tailscale serve` - (never `funnel`). If any of that isn't true, fall back to SSH and say so. -- **Safety, not just reachability:** home base trusts the connection itself as the - boundary, so the link must stay private — a private SSH tunnel or the tailnet via - `serve` — and never a publicly exposed port (`funnel`, or a public bind). +- Before relying on Tailscale, verify both machines are online in `tailscale + status` and the peer's MagicDNS name resolves. Do not ask the user to expose a + port publicly, use `tailscale funnel`, or configure router port-forwarding. **Per-OS reality of the steady link** (the machine being added is what matters): @@ -142,13 +144,12 @@ amico fleet enroll --join-token ~/.amico/ops/fleet/join-token.json \ The verb runs the whole thing: build check → link config → register → install the guard/link → confirm. Read its JSON verdict (Step 6a). -**Its own studio (peer).** There is no turnkey wizard yet — guide it: on the new +**Its own studio (peer, recommended).** There is no turnkey wizard yet — guide it: on the new machine run **Amicode: Fleet — Go Standalone** (it runs its own engine), then have it advertise itself (`serving`) and point the directory pointer (`~/.amico/ops/fleet/keeper.json`) at the roster host so both machines see each -other with an **Attach** control. Repeat the honest caveat: attaching flips the -pointer, but running your live session *on* the other machine's engine is still -landing. +other with an **Attach** control. Explain that each machine runs its own engine +and session activity and prompts sync between machines in near real time. ### Step 6a — read the verdict, but trust the real link diff --git a/packages/extension/skills/create-a-fleet/SKILL.md b/packages/extension/skills/create-a-fleet/SKILL.md index b61b3e5c6..11128752f 100644 --- a/packages/extension/skills/create-a-fleet/SKILL.md +++ b/packages/extension/skills/create-a-fleet/SKILL.md @@ -32,7 +32,7 @@ terms live in the commands you run, never in what you say: | hub / star / server / canonical | "home base" — the machine that hosts your work | | client / thin client | "a window onto home base" | | standalone | "works on its own" | -| peer studios | "two independent studios you hop between" | +| peer studios | "your own studio on each machine, with sessions and prompts syncing between them" | | enroll / join token | "link this machine in" | | verify-attach / roster / health | "check the link actually works" | | transport (ssh / tailscale) | "how the machines reach each other" | @@ -72,21 +72,18 @@ Ask it in outcomes, never with "hub" or "peer": > **"When your machines are apart — one's off, or you've taken the laptop out — > what should happen?"** -> - **"One machine hosts my work; the others are windows into it"** (recommended) -> → home-base setup. The fully-wired path: your sessions live on home base and -> follow you to any machine. -> - **"Each machine should fully work on its own, and I'll hop between them"** → -> independent studios. - -**Default to home base with the always-on machine as home base** (the workstation, -not the laptop) — it's the coherent, fully-wired experience. Pick "independent -studios" only when the user explicitly wants each machine usable alone, and then -give the honest caveat up front: *"You can set that up today and hop between them, -but running a live chat directly on the other machine's engine is still landing — -so the smoothest 'both as one' is still home base. Independent anyway, or home -base?"* Honor their choice; jump to **Peer studios** below for that path. - -The rest of this skill is the home-base (hub/star) build. +> - **"Each machine should work on its own, while my fleet sessions stay in sync"** (recommended) → peer studios. Each machine has its own engine, and session activity and prompts sync between machines in near real time. Any one machine can keep working if another is offline. +> - **"I want one machine to host everything; the others are windows onto it"** → home-base setup. The other machines depend on home base being reachable. + +Default to peer studios when the user wants multiple machines to work as one fleet +without a single machine being a point of failure. Explain that each machine runs +its own engine, while session activity and prompts sync between them in near real +time. Choose home base when the user specifically wants one shared engine/database +and accepts that the other machines depend on it. Honor their choice. + +If the user chooses peer studios, skip the home-base provisioning and window-link +steps below; follow **Peer studios** instead. The remaining numbered steps describe +the home-base (hub/star) build. ## Step 2 — discover the machines @@ -160,11 +157,14 @@ On failure, walk **one** step in plain terms and retry — don't recite a pipeli One machine at a time. While one is paused awaiting a human step, move to the next. -**a. How it connects (transport).** Default **SSH** (the universal floor, works -everywhere). Choose **Tailscale** only for a machine that roams *and* only after -verifying both machines are online in `tailscale status`, MagicDNS resolves, and -home base shares its engine via `tailscale serve` (never `funnel`) — else fall back -to SSH and say so. +**a. How it connects (transport).** Recommend **Tailscale** for machines on +different or changing networks: it creates a private connection without router +port-forwarding. If Tailscale is not installed or signed in, point the user to +[Tailscale's install guide](https://tailscale.com/kb/1017/install) and pause that +machine until setup is complete. Verify both machines are online in `tailscale +status` and that the peer's MagicDNS name resolves before using it. Use **SSH** +when the machines are on a trusted reachable network or the user prefers SSH. Never +ask the user to expose a fleet port publicly or configure router port forwarding. **b. Confirm.** Show the exact link-in for this one machine; get an explicit yes. @@ -229,34 +229,34 @@ you'll see the same sessions."** Name any machine still paused and the one step it's waiting on — nothing is silently dropped. Anything still stuck after its guided step is a job for `/troubleshoot-fleet`; hand off and stop. -## Peer studios (the independent-studios path) +## Peer studios (the recommended path) -The additive topology: every machine runs its **own** engine on its **own** -database and *advertises* itself; you attach/switch between them. A machine that's -off doesn't strand the others — there is no single machine whose loss stops the -fleet. A peer stays "on its own" (never a window onto home base) and never installs -the never-fork guard. +Every machine runs its **own** engine on its **own** database and advertises itself. +Session activity and prompts sync between machines in near real time, so users can +work from either machine without making one machine the host for all work. If a +machine is offline, the others keep working. A peer remains standalone (never a +window onto home base) and does not install the never-fork guard. Do not describe +peer studios as merely switching between isolated histories. **What works today — and the honest edges (state it plainly):** - **Works:** each machine as its own independent, resilient studio; advertising - (`serving`); the shared directory of machines (the keeper); the **Attach** control - and per-machine transport/credential plumbing; **observing** another machine's - sessions (read) works without a grant. + (`serving`); the shared directory of machines (the keeper); synced session + activity and prompts; the **Attach** control and per-machine transport/credential + plumbing. - **Sharp edge — observe hands a write-capable credential:** *controlling* another machine's session (prompting/archiving it) is wired, but the machine you attach to currently accepts the same credential it accepts for *reading* — so granting someone "observe" effectively grants write too. A separate control-only credential is future work. Scope who you grant to accordingly. -- **Not yet:** clicking **Attach** does not yet run your *live* session on the other - machine's engine (the live re-target is a scoped follow-up). So peer is great for - independent + resilient studios today; live shared driving across them is landing. - Per-machine removal is `amico fleet revoke ` (drops the grant + bars re-mint + fans out to serving peers); verify a revoked machine can no longer reach the others rather than trusting the exit code. -**No turnkey peer wizard exists yet**, so guide it by hand. To make an existing -home-base/window pair into independent studios: +**No turnkey peer wizard exists yet**, so guide it by hand. For a new peer fleet, +keep each machine standalone, have each advertise `serving`, and point each +machine's `~/.amico/ops/fleet/keeper.json` at the shared directory host. For an +existing home-base/window pair: 1. On the **window machine**, run **Amicode: Fleet — Go Standalone** — it now runs its own studio and no longer depends on home base. diff --git a/packages/extension/skills/troubleshoot-fleet/SKILL.md b/packages/extension/skills/troubleshoot-fleet/SKILL.md index a10806ee5..c63b980c0 100644 --- a/packages/extension/skills/troubleshoot-fleet/SKILL.md +++ b/packages/extension/skills/troubleshoot-fleet/SKILL.md @@ -32,7 +32,7 @@ never in what you say: |---|---| | hub / server / canonical | "home base" — the machine that hosts your work | | client / thin client | "a window onto home base" | -| standalone | "working on its own" | +| standalone peer | "its own studio, syncing fleet sessions with the other machines" | | tunnel | "the steady link between your machines" | | guard | (don't mention it — it's plumbing) | | verify-attach / roster / health | "whether the link actually works" | @@ -68,8 +68,11 @@ accidentally test the wrong machine and call a broken link healthy. When you run a cross-machine probe, run it inside `ssh` and keep the command you used so you can show your work if asked. +- If this is a **peer fleet**, each machine has its own engine and session activity + and prompts sync between machines. Test the affected peer's own configured + connection; do not assume every machine routes through one home base. - If the fleet has a **home base** (a `server`) and you're on it, the other - machines reach you through the link — to test *their* side you must `ssh` into + machines reach it through the link — to test *their* side you must `ssh` into them. - The fleet port is almost always **4096**. Read the real one from `~/.amico/ops/fleet/fleet.json` (`canonical.port`) before assuming. @@ -123,7 +126,7 @@ traps, both real in the current build: | Boot warning "fleet issue"; guard/settings/link drift | "Your machine's link setup drifted out of sync." | **Fleet — Repair** (re-installs the link + settings from source; idempotent). | | Sessions look wrong / a machine seems to have its *own* private history | Likely a silent fork — see **Step 3**. | Detect the fork first; do NOT reload blindly. | | Version/pin mismatch on join | "The two machines are on different Amicode builds." | Get both on the same build (**Repo Sync**, then rebuild/reload), then re-join. | -| "attached server unreachable" / a peer won't load | "The other machine isn't reachable right now over the way it's set to connect." | Check that peer is on and reachable by its set method; re-probe its real endpoint (Step 1). This is an honest "it's off," not a bug. | +| A peer is missing, unreachable, or its sessions stop syncing | "That machine isn't reachable over the private connection the fleet uses." | Check that peer is on and reachable over its configured connection; for Tailscale follow Step 3, then re-probe its real endpoint. | | Approval prompts on every fleet action | "VS Code is asking permission for each fleet file it touches." | It's a one-time settings fix; offer to walk it (allow the external directories in the opencode config), then it stops. | ## Step 3 — transport & OS-specific failures (the cross-machine gotchas) @@ -131,19 +134,23 @@ traps, both real in the current build: The link is set up differently per machine, so the fix depends on the machine's OS and how it connects: -- **SSH link down (the common case).** The steady link is an SSH forward. If it +- **Peer connection down.** For peer fleets, machines connect directly using their + configured transport; there is no single home-base tunnel. If using SSH, confirm + the alias still resolves and reaches the peer. If using Tailscale, confirm both + machines are online and MagicDNS resolves. Restore that private connection and + verify session activity and prompts sync again. +- **Home-base SSH link down.** The steady link is an SSH forward. If it won't stay up: confirm the SSH alias still resolves and reaches the machine (`ssh -o BatchMode=yes -o ConnectTimeout=6 'echo ok'`), and that home base has Remote Login (macOS) / sshd (Linux) on. Re-establish with **Fleet — Repair**. -- **Tailscale link down.** If a machine connects over Tailscale, `tailscale - status` must show *both* machines online and MagicDNS must resolve, and home - base must be sharing its engine over the tailnet with `tailscale serve` — **not** - `tailscale funnel`. This matters for safety, not just reachability: home base - trusts the link itself as the boundary, so it must stay on the private tailnet - (`serve`) and never be exposed to the public internet (`funnel`) or bound to a - public address. If Tailscale is down, the honest move is to fall back to the - SSH link. +- **Tailscale link down.** Tailscale is the recommended way to connect machines + across networks without router port-forwarding. Check `tailscale status` on both + machines and confirm MagicDNS resolves. If Tailscale is not installed or signed + in, send the user to [Tailscale's install guide](https://tailscale.com/kb/1017/install). + Keep fleet traffic private: never use `tailscale funnel`, expose a public bind, + or ask the user to configure router port-forwarding. SSH is an alternative when + it is already reachable and the user prefers it. - **Linux / WSL window-onto-home-base machine — the known gap (#1260).** On macOS the steady link is a managed background service that auto-reconnects. **On Linux and WSL that managed link is not wired yet (#1260)** — the app installs the @@ -160,11 +167,11 @@ OS and how it connects: ## Step 4 — silent fork detection (why "reload" is not always safe) -The whole point of a window-onto-home-base machine is that it **never runs its own -copy of the engine** against a private database — two live writers on one database -is the corruption case. When a window machine silently spawns its own engine, it -grows a private history and the two machines quietly diverge. Detect it before you -tell anyone to reload: +The fork check applies to a **window onto home base**, which must not spawn its own +engine. Do not apply this diagnosis to peer studios: each peer is supposed to run +its own engine, while session activity and prompts sync between machines. When a +window machine silently spawns its own engine, it can develop a private history. +Detect that specific failure before telling anyone to reload: ```bash # on the suspect window machine (ssh in if needed), on the fleet port: @@ -178,16 +185,17 @@ ssh 'curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:/se # 401 on a window machine = an armed LOCAL fork; home base answers 200 (anonymous). ``` -If you find a fork, the fix is to stop the fork, restore the guard + link +If you find a fork on a home-base window, the fix is to stop the fork, restore the guard + link (**Fleet — Repair**), and — if the forked copy grew real history the user cares about — hand off to `/fleet` for the recovery, which knows how to merge a stranded database safely. **Never** "just reload" over a fork: you would either lose the forked history or keep two writers alive. -Home base itself keeps its shared database at `~/.amico/server/session.db` and -runs as the `co.harmoniqs.amico-hub` service (a launchd agent on macOS, a -`systemd --user` unit on Linux). If home base's history looks empty or wrong, -that is a home-base problem, not a window-machine fork — go to `/fleet`. +Home base keeps its shared database at `~/.amico/server/session.db` and runs as +the `co.harmoniqs.amico-hub` service (a launchd agent on macOS, a `systemd --user` +unit on Linux). Peer machines have their own engines; fleet session activity and +prompts sync between them. If sessions stop syncing across peers, diagnose peer +reachability and transport before treating it as a database fork. ## When to escalate to the internal `/fleet` playbook diff --git a/tools/fleet/README.md b/tools/fleet/README.md index 9aa7e393d..363e61533 100644 --- a/tools/fleet/README.md +++ b/tools/fleet/README.md @@ -103,9 +103,10 @@ substrate for it. Horizon 2 itself is **declared, not built** (`compute` stays i ### What a peer is -- **Independent studio.** Each peer runs its own engine on its own DB (single-writer-per-DB - preserved *per machine* by the existing adopt-or-spawn gate). If one peer is off, the others - keep working — **there is no single engine whose loss stops the fleet.** This is the key +- **Independent studio with synced fleet sessions.** Each peer runs its own engine on its own DB + (single-writer-per-DB preserved *per machine* by the existing adopt-or-spawn gate), while + session activity and prompts sync between machines in near real time. If one peer is off, the + others keep working — **there is no single engine whose loss stops the fleet.** This is the key difference from a hub client, which is dead whenever the hub is down. - **Standalone, never a client.** A peer keeps `role: standalone` (or no `fleet.json`) and *advertises*; it never writes `role: client` and never installs the never-fork guard. @@ -119,25 +120,16 @@ substrate for it. Horizon 2 itself is **declared, not built** (`compute` stays i | Directory keeper pointer | `keeper_pointer.ts` → `~/.amico/ops/fleet/keeper.json` (env `$AMICO_FLEET_KEEPER_FILE`) | A resolvable `{sshAlias, transport}` coordinate for the roster host, carried in its own file (never derived from the `roster.json` it hosts — that would be circular). | | Switch-control pointer | `attachment_pointer.ts` (env `$AMICO_FLEET_ATTACHMENT_FILE`); honesty surface `GET /amicode/fleet/attachment` | The "currently attached server". Empty ⇒ local. Never proxied. | | Attach / detach | `POST /amicode/fleet/attach {machine_id, base_url?, token?}` and `POST /amicode/fleet/detach` (registered **unconditionally** — a standalone peer must attach) | Adds/removes an upstream and drives the pointer; the roster is the sole candidate source. A switch resets the SSE cursor. | -| Per-attachment transport + credential | `attachment_transport.ts`, `attachment_credential.ts` | SSH is the universal default; `roaming` → tailscale, direct opt-in. Each attach injects a UI client credential to authenticate to the peer engine. | +| Per-attachment transport + credential | `attachment_transport.ts`, `attachment_credential.ts` | Tailscale is recommended for machines on different networks; SSH works when peers are already privately reachable. Each attach injects a UI client credential to authenticate to the peer engine. | | Fleet Manager **Attach** control | overlay `packages/app/src/pages/session/fleet-manager.ts:219` | The per-device-row Attach/Detach button — the front-end that drives the attach route. | | Placement seam | `amicode_session` / Task dispatch `placement` field, default `local` | Threaded and defaulted for Horizon 2; **provably inert** today (nothing routes on it). | -### The honest limitation — live re-target is NOT wired yet +### Shared sessions across peers -This is the one caveat to state plainly, and the reason a peer setup does not yet fully -replace a hub for *shared* work: **`server.ts` is held byte-identical, so the three-way -resolver (`resolveAmicodeTarget`, `attachment_pointer.ts`) is not in the live proxy's -`dispatch()`.** An attach flips the switch-control pointer and resets the SSE cursor, but the -**live proxy/SSE still target the local (or hub) engine** — "switch → now driving the peer's -live engine end-to-end" needs the `server.ts` invariant relaxed in a scoped, reviewed edit -(a flagged follow-up from #1353). So today: - -- **Works:** each peer as an independent, resilient studio; advertising; the roster/keeper - directory; the Attach control and pointer flip; the credential/transport plumbing; - everything at the unit/route level. -- **Not yet:** clicking Attach and having your live session actually run on the other peer's - engine. +Peer machines keep their own engines and databases, while session activity and prompts sync +between them in near real time. Users can work with fleet sessions from either machine without +making one peer the sole host. The Attach control selects a peer; it does not merge the peers' +local databases into one shared database. ### Making two hub machines into peers (no single point of failure) @@ -155,8 +147,13 @@ machine is a single point of failure: keeper bootstrap pointer (`~/.amico/ops/fleet/keeper.json`) to the roster host, so both show up in the Fleet Manager and expose the Attach control. +For peers on different networks, use Tailscale for private reachability rather than asking users +to configure router port-forwarding. Direct users to +[Tailscale's install guide](https://tailscale.com/kb/1017/install) when needed. Never expose +fleet services with `tailscale funnel` or a public bind. + After step 1–2 the single-point-of-failure is gone (each machine has its own engine); step 3 -adds mutual discovery + the Attach scaffolding, with the live hop pending the caveat above. +adds mutual discovery. Session activity and prompts sync between the peers when they are online. ## Prevention