feat(fleet): run one command, or a package upgrade, across many servers

[ralyodio]

DiskPush moves bytes to a server. This adds the other thing you do with a list of servers: run work on all of them.

diskpush fleet check   --on tag:production          # what does each one need
diskpush fleet upgrade --on tag:production --sudo   # install it
diskpush fleet run "systemctl reload nginx" --on 'web-*' --sudo
diskpush fleet script ./deploy.sh --on all '!db-01'

In the desktop app it is the Fleet button in the header: tick servers on the left, pick a recipe or type a command, watch each host report on its own.

What's here

• packages/fleet-core — new package, no Electron in it, the same way rsync-core has none. Takes connections and a script, produces a stream of per-host events. It opens sessions only through a connect function the caller supplies, so the CLI opens one connection per host and closes it while the desktop hands it a pooled session it keeps — and so the runner is testable without a network.
• diskpush fleet — run, script, upgrade, check, servers, commands, runs, show.
• Desktop Fleet dialog — server list with tag filter, recipe picker, per-host live output cards, and the read-only update sweep.
• Six built-in recipes — check-updates, upgrade, reboot-required, disk, uptime, who. They can be copied but not edited, so upgrading DiskPush never silently changes a command someone relies on.
• Migration 002-fleet — saved commands, runs, and per-host results.
• SshSession.execStream — exec buffers to completion, and a fifteen-minute upgrade is only watchable if the output arrives while it happens.
• docs/fleet.md, plus updates to the README, CLI reference, desktop, security and architecture docs.

Selecting servers

--on takes names, globs, tag:NAME, host:GLOB, all, and !TERM to exclude. Includes are unioned then exclusions subtracted, so order does not matter. Saved connections and ~/.ssh/config hosts are both selectable.

Upgrades on a fleet that is not all the same

The package manager is detected on each host, inside the script, so one command covers Debian, Rocky and Alpine together — apt, dnf, yum, zypper, pacman, apk, brew, pkg. Every invocation is non-interactive and answers "keep the installed config file" where the question comes up, because a fleet upgrade that stops on a conffile prompt on host four has already failed. Nothing is removed: upgrade, never dist-upgrade or autoremove. Rebooting is off by default; the run reports which servers need one.

The decisions worth reviewing

• Script text is never interpolated into a command line. It goes to the remote interpreter on stdin; the command line only names the interpreter. A quote, a backtick or a newline in a script cannot become a different command.
• A selector term matching nothing is an error, not a smaller fleet. --on web-O3 with a letter O fails rather than quietly patching eleven of twelve servers. Same for an exclusion, which was meant to protect a host.
• unreachable is not failed. A server that was switched off never ran the command. Collapsing those two is how a fleet tool reports a powered-down box as a failed deploy.
• Destructive patterns need confirming — the same bargain Mirror makes on the transfer side. Re-checked in the Electron main process, so a renderer that skipped the dialog cannot skip the check with it. It is a tripwire against the accident, not a sandbox against an adversary, and the docs say so.
• --sudo uses sudo -n, which fails rather than hanging on a prompt nobody can see. --sudo-password asks once without echo and feeds sudo -S on stdin: never stored, never logged, never on a command line where ps on the server would show it.
• Unknown host keys fail during a fan-out instead of prompting one at a time; --accept-new is opt-in per run. A changed key is still never accepted.
• A run records the script it ran and the server names it ran on, not pointers to them, so editing a saved command cannot rewrite the history of what was executed on production last Tuesday.
• Exit 71 when any server did not succeed. One code rather than a failing host's own status: across twelve servers there may be several, and picking one would mean inventing a winner. --json carries them all.

Verification

Tests: 486 across 40 files (was 416/35). 70 new, covering selection, the hazard guard, command building and quoting, upgrade-script generation and output parsing, the runner (concurrency bounds, abort, failed/unreachable/timeout), fleet persistence, the IPC contract, and the renderer's event reducer.

Two of those test files exist because reading the code was not enough:

• recipes.test.ts runs every read-only recipe for real under /bin/sh -es and asserts exit 0. upgrade is excluded there because it installs packages.
• guard-calibration.test.ts runs ten realistic scripts (a node deploy, an nginx reload, docker cleanup, log rotation, a pg backup, cert renewal, a scoped chown, a reboot-required check, read-only firewall status, a disk check) and asserts the guard stays silent, plus ten genuinely dangerous ones and asserts it fires. Currently 0 false positives, 0 misses — either number moving is the signal.

Bugs found while verifying

Eight, six of which were only visible by running the thing rather than reading it:

1. The --sudo-password prompt echoed the password. The widely-copied readline _writeToOutput recipe for hidden input silently does not work on current Node. Only visible under a pty; piping into it takes the isTTY guard instead. Rewritten with raw mode.
2. sh -e made the status probe report healthy servers as unreachable — grep -c exits 1 when it counts zero updates. The workaround only fixed one of the five ways to run that script, so the fix moved into the script itself as set +e.
3. reboot-required failed on the branch that exists to say "yes, reboot this one" — [ -f pkgs ] && cat pkgs as the last statement of the then block makes its status the if's status, so a server needing a reboot with no package list reported as failed.
4. The hazard guard fired on reboot=no and echo "reboot required" — DiskPush's own read-only recipe demanded confirmation to read a file.
5. Modern sudo says "interactive authentication is required", not "a password is required", so the one actionable error message never appeared on current Ubuntu.
6. --print-command polluted stdout with a three-line preamble, and --reboot --print-command stopped to ask a question nobody was there to answer.
7. A bad selector exited 70 (internal) instead of 65, and bypassed JSON error output under --json.
8. A doc described the check table's placeholder as an em dash where the CLI renders a hyphen.

Also verified

Live against a real SSH server: streaming interleaves across hosts as it happens, per-host exit codes and timeouts are reported correctly, script bytes round-trip unmangled through quotes / backticks / $VAR / pipes / redirects, an unreachable host stays distinct from a failing one, --stop-on-error leaves queued hosts untouched, and every shipped recipe runs clean with no flags.

The desktop smoke test passes with the Electron main process actually loading the new ESM dependency — the failure mode that script was written for.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UeSWg1Czsb2Lwxj8vHUnA4