feat(security): preflight audit setup and define complete baseline evidence

CHANGELOG.md

@@ -17,6 +17,9 @@ Every entry says what changed as a caller sees it, why, and what to watch for 
 
 ### Added
 
+- `security preflight` checks scanner pins, command availability, canonical repository paths and
+  aliases, GitHub analysis access and an optional comparison baseline without scanning source.
+  JSON diagnostics distinguish scanner readiness from the separately approved audit stages.
 - Drive AI extraction research drives the executable against a guarded loopback responder,
   checking evidence quotes, proposal persistence, provider failures and source changes.
   Historical npm and local candidate results are explicitly distinguished.

README.md

@@ -26,7 +26,8 @@ Install this package as a development dependency and build it before running fro
 Give every scan a new private output directory; a recorded stage is never overwritten.
 
 ```sh
-cli-testing security scan <run-dir> <checkout>...
+cli-testing security preflight <checkout>... [--previous <run>]
+cli-testing security scan <run-dir> <checkout>... [--previous <run>]
 cli-testing security socket <run-dir> <checkout>...
 cli-testing compare security <run-a> <run-b>
 ```

docs/dev/ARCHITECTURE.md

@@ -11,6 +11,7 @@ Published as `@wirecat/cli-testing`, a development dependency of max-cli and tg-
 | [`src/leaks.ts`](../../src/leaks.ts) | bounded text-artifact and diagnostic leak checks |
 | [`src/contract-main.ts`](../../src/contract-main.ts) | the `cli-contract` executable |
 | [`src/runner.ts`](../../src/runner.ts) | pinned scanner stages and the explicitly live agent plan |
+| [`src/preflight.ts`](../../src/preflight.ts) | read-only tooling, canonical repository and comparison-baseline checks |
 | [`src/runs.ts`](../../src/runs.ts) | private run layout, immutable stage records and legacy reads |
 | [`src/observations.ts`](../../src/observations.ts) | scanner candidates reduced to stable identifiers |
 | [`src/compare.ts`](../../src/compare.ts) | observation history and transcript redaction |

docs/dev/BACKLOG.md

@@ -19,8 +19,9 @@ the private `cli-private` repository, never here.
 
 ## Phase 1 — remaining security work
 
-- **SEC-5 · P2** A second audit run of max-cli and tg-cli after the first audit's fixes ship,
-  compared with the first; the agent suite on tg-cli for the first time.
+- **SEC-5 · P2** A complete new audit over the current repository set, with explicit scanner,
+  Socket, manual/false-positive and owner-approved agent stages. Start a fresh scanner baseline
+  if historical coverage is incomplete; compare only complete records with matching coverage.
 
 ## Phase 2 — contract
 

docs/security/LESSONS.md

@@ -35,6 +35,13 @@ Facts learned during the audits, with their source. Each cost time once.
 - **Trail of Bits `fp-check` installs a Stop hook on every session** that blocks stopping until the
   verification is visible in the conversation itself. Disabling the plugin takes effect from the next
   session; in the current one, turn it off in `/hooks`.
+- **Zero scanned history is not a clean secret scan.** Gitleaks splits `--log-opts` itself;
+  embedded shell quotes can select zero commits. Keep the history/path expression in one shell
+  argument without internal quotes, verify a nonzero scanned count and use a separate synthetic
+  known-positive control. A private allowlisted scan does not establish whole-repository coverage.
+- **A delivery closeout is not a new audit baseline.** Do not change historical incomplete runs
+  to pass comparison. Bootstrap a new scanner run without `--previous`, record its coverage and
+  track manual verification, Socket and live A7 independently.
 
 ## GitHub
 

docs/security/METHOD.md

@@ -20,6 +20,11 @@ gh repo list WireCatLabs --limit 100 --json name,visibility --jq '.[] | "\(.name
 
 ## A run, step by step
 
+Start with the read-only preflight in [RUNBOOK](RUNBOOK.md#preflight-and-baseline). Use a new
+baseline without `--previous` when no complete comparable run exists. Historical gaps stay gaps;
+scanner completion never substitutes for the manual/Socket/A7 stages. The
+[regression checklist](REGRESSIONS.md) defines the evidence and stage ledger for a complete audit.
+
 1. `cli-testing security scan <run-dir> <checkout>... [--previous <run>]` creates the run folder,
    refuses tool versions other than the pins (or pins younger than two weeks), and calls
    [`scripts/security/scan`](../../scripts/security/scan). The script exports `origin/main` of each
@@ -33,9 +38,10 @@ gh repo list WireCatLabs --limit 100 --json name,visibility --jq '.[] | "\(.name
    adds its exit code to `run.json`. The Socket CLI banner prints the start of the API token; the
    script passes `--no-banner` after the command (before it, the flag is not parsed). Raw output
    goes to `<run-dir>/raw/<repo>/<tool>.json`, so two runs can be diffed.
-3. gitleaks over the full history of every repository, `cli-private` included, from each
-   repository's root (it reads `.gitleaksignore` from the current folder):
-   `gitleaks git --no-banner --redact .`. Exit codes and hit counts go into `SCOPE.md`.
+3. gitleaks over the full history of each public repository, from its root (it reads
+   `.gitleaksignore` from the current folder): `gitleaks git --no-banner --redact .`. The private
+   allowlist/exclusions must respect the owner's file permissions and are recorded separately;
+   see RUNBOOK. Record exit codes, hit counts and nonzero scanned history in `SCOPE.md`.
 4. Read the GitHub settings that `scan` does not (commands in "GitHub settings") into `SCOPE.md`.
 5. Do the manual review (section "Checklist"). Every item gets a verdict in `runs/<date>/REPORT.md`:
    safe, finding, or not applicable — each with the evidence that supports it.

docs/security/REGRESSIONS.md

@@ -0,0 +1,56 @@
+# Audit regression checks
+
+Use this when preparing or repeating an audit after fixes. It helps reviewers prove that the
+selected protection works, ordinary supported workflows still work, and incomplete stages remain
+visible. Read the owner's accepted-risk decisions before selecting controls; this checklist does
+not authorize account access, settings changes, publication or live messenger actions.
+
+## Record the boundary
+
+For each selected fix, record privately: finding identifier, source commit, released package and
+exact dependency pins, owning test, executed command, positive and negative results, limitations
+and reviewer. Test current source and the installed release separately when they differ. A test
+name or green unrelated CI run is not proof that the changed behavior was exercised.
+
+Use synthetic accounts/data and temporary state for process tests. Never open the owner's store,
+read credential files, install a real user timer or send a messenger message as a routine PoC.
+
+## Protection and compatibility controls
+
+| Area | Protection control | Allowed-workflow control |
+| --- | --- | --- |
+| Source-analysis gate | A synthetic critical SARIF result blocks; missing/unresolvable rule metadata fails | A lower-severity result is reported under the approved policy; resolve rules in driver and extension components |
+| Scheduled authority | Under the inherited read-only lock, send and authority-granting config/permissions/recipients/schedule changes refuse before execution | Allowed reads and reply-text preparation work; do not claim the CLI lock sandboxes other programs |
+| Credential paths | Synthetic canonical, symlinked and platform-case variants of protected paths refuse before bytes are read or transferred | Ordinary hidden working folders remain allowed; exercise only the explicitly supported override |
+| Parser complexity | A bounded adversarial fixture completes inside a measured resource budget or fails with the defined error | Ordinary valid HTML/documents preserve expected text; accepted long-document behavior stays supported |
+| MCP text boundaries | Incoming text is framed as data and hidden controls are visible where the released contract promises it | Original Unicode writes and ordinary machine JSON follow the accepted compatibility policy |
+| File creation | A pre-existing target/symlink refuses an exclusive temporary/output write and retains sentinel contents | A new target succeeds; normal valid output is not truncated or overwritten |
+| Deployment/publishing | Actual secret-name metadata and environment branch rules match the intended credential boundary | The authorized main deployment/publisher succeeds with those scoped credentials; workflow text alone is insufficient |
+| Audit tooling | Wrong pins, missing source analysis, failed scanners and incomplete/mismatched comparison coverage remain failures or explicit gaps | Canonical symlinks work, supported scanner exit codes retain observations, and a new baseline needs no invented previous run |
+
+For secret scanners, use a separate temporary synthetic Git history with a known-positive fixture.
+Require a detection, then verify that the intended clean control scans nonzero commits. Record the
+actual allowlist and exclusions. Do not copy canaries into maintained source or publish findings.
+
+## Completion ledger
+
+Keep this table in the private run report, with a source/artifact link for every status. Use
+`run`, `not run`, `failed`, or `not applicable` with a reason. A successful CLI `scan` means only
+that scanner stage completed; it does not fill the other rows.
+
+| Stage | Required evidence |
+| --- | --- |
+| Preflight | Tooling commit, paths/aliases, tool pins, source and GitHub access, baseline choice |
+| Static scanners | Source commits, versions, exit codes, coverage/ignore inventory and retained parser errors |
+| CodeQL | Analysis availability, open-alert snapshot and manual triage; 404 is not zero alerts |
+| Secrets | Public full-history results; private allowed history scope; positive control and nonzero scanned history |
+| Socket | Approved source-upload scope, pinned CLI, same commits, valid successful report; owner-terminal authentication |
+| Manual review | Assigned S1–S10/A1–A7 owners, accepted-risk input, source/release boundary and evidence per verdict |
+| False-positive verification | Bounded PoC and negative control, reviewer verdict and limitations before dismissal |
+| Agent A7 | Explicit approval for cast/payloads, host-refused canary, utility checks, attempted writes and verified cleanup |
+| Comparison | Complete comparable scanner records with matching repository/scanner coverage; missing coverage cannot become fixed observations |
+| Delivery follow-up | Fix PR, exact published consumer/dependency versions, targeted regression evidence and updated private register |
+
+An audit is complete only when every required row has evidence, or an explicit approved
+not-applicable decision. `not run` and `failed` rows keep the audit partial. Keep historical
+reports unchanged; record later delivery in a linked closeout and the current findings register.

docs/security/RUNBOOK.md

@@ -1,6 +1,7 @@
 # Security audit — runbook
 
-The commands of one audit run, in order. Why each step exists: [`METHOD.md`](METHOD.md). The
+Use this before starting an audit to prepare the checkouts, validate the tools and record each
+stage without overstating coverage. Why each step exists: [`METHOD.md`](METHOD.md). The
 first run was 2026-10-09; its results are private (`cli-private/security/runs/2026-10-09/`).
 
 ## 0. Tools, pinned
@@ -9,15 +10,21 @@ A version at least two weeks old on the day of the run, recorded in `SCOPE.md` b
 `cli-testing security scan` refuses any other version.
 
 ```sh
-uv tool install semgrep==1.178.0
-uv tool install zizmor==1.30.1
+AUDIT_TOOLS="${XDG_CACHE_HOME:-$HOME/.cache}/wirecat-audit/semgrep-1.178.0"
+test -x "$AUDIT_TOOLS/bin/python" || uv venv "$AUDIT_TOOLS"
+uv pip install --python "$AUDIT_TOOLS/bin/python" semgrep==1.178.0 zizmor==1.30.1
+export PATH="$AUDIT_TOOLS/bin:$PATH"
+export SEMGREP_SEND_METRICS=off SEMGREP_ENABLE_VERSION_CHECK=0
 # gitleaks: the release binary, checked against its checksums file the same way as osv-scanner
 # osv-scanner: the release binary, checked against the release's own SHA256SUMS
 V=2.6.0; D=$(mktemp -d); cd "$D"
 gh release download v$V -R google/osv-scanner -p osv-scanner_linux_amd64 -p osv-scanner_SHA256SUMS
 grep ' osv-scanner_linux_amd64$' osv-scanner_SHA256SUMS | sha256sum -c - && install -m 755 osv-scanner_linux_amd64 ~/.local/bin/osv-scanner
 ```
 
+Keep the audit environment first on PATH for the whole run: Semgrep also starts `osemgrep`
+through PATH. This preserves any different global installation.
+
 Socket CLI runs through `pnpm dlx @socketsecurity/cli@1.1.179` inside the script. Log in once,
 **in your own terminal**, never through an agent: `pnpm dlx @socketsecurity/cli@1.1.179 login`.
 
@@ -26,7 +33,57 @@ Claude Code plugins: `claude plugin marketplace add trailofbits/skills`, then in
 fp-check agentic-actions-auditor differential-review` (`claude plugin install <name>@trailofbits`).
 Disable `fp-check` again after the run — its Stop hook fires on every session.
 
-## 1. GitHub settings — apply, then read back
+## Preflight and baseline
+
+Run from a clean cli-testing worktree built from the commit you intend to use. This does not
+require a globally installed executable:
+
+```sh
+P="$HOME/Projects/AI"
+TESTING="$P/cli-testing"
+PRIVATE="$P/max-cli/docs_ai"
+RUN="$PRIVATE/security/runs/$(date +%F)"
+C=("$P/max-cli" "$P/tg-cli" "$P/cli-core" "$P/cli-base" "$P/cli-memo" "$P/drive-cli" "$P/zoom-cli" "$P/cli-docs"
+  "$P/cli-testing" "$P/community" "$P/wirecat-github")
+cd "$TESTING"
+pnpm install --frozen-lockfile
+pnpm build
+node "$TESTING/dist/main.js" security preflight "${C[@]}"
+```
+
+Set `TESTING` to the clean tooling worktree and `PRIVATE` to the private checkout in use. Verify
+that the output directory is inside that private checkout. The public source repositories are
+exported from `origin/main`; unrelated local edits are not scanned. Fetch the intended refs before
+preflight. Record both source commits and the cli-testing commit in the new scope.
+
+A local `cli-core` symlink to `cli-messaging` is supported: keep the canonical `cli-core` name in
+`C`, and do not include both aliases. Preflight rejects duplicate physical checkouts, unrecognized GitHub
+origins, private source repositories, unavailable commands,
+unsupported scanner pins, unreadable CodeQL analysis and invalid comparison metadata. It only
+reads Git/GitHub metadata and tool versions; it does not scan source, create a run, read credential
+files, log in, change settings or authorize live tests. Socket authentication remains a separate
+owner-terminal check; preflight does not start Socket or fetch a token.
+
+For the first valid scanner baseline, omit `--previous`. An older report may still be used as
+manual historical evidence; never change its `complete` flag or manufacture missing outputs.
+For a later comparison, set an existing complete baseline explicitly:
+
+```sh
+PREVIOUS=/absolute/path/to/a/completed-private-run
+node "$TESTING/dist/main.js" security preflight "${C[@]}" --previous "$PREVIOUS"
+```
+
+The baseline needs a complete security `run.json` and recorded coverage. `compare` additionally
+requires identical repository/scanner coverage. A missing analysis, failed parser or absent stage
+is a gap, not a fixed finding. A CodeQL 404 may mean absent analysis or insufficient access; resolve
+it or record the repository as incomplete. Never replace it with an empty alerts array. A
+successful scanner stage is not a complete METHOD audit: track manual review, false-positive
+verification, Socket, full-history secret scanning and A7 separately in the report.
+
+## 1. GitHub settings — read back, then apply approved changes
+
+First read settings with the commands in METHOD. An audit does not authorize settings changes.
+Apply the following only when the owner has approved the specific correction, then read it back.
 
 New public repositories get their settings from the organisation configuration `WireCat public`;
 check that it is still the default with the command in [`METHOD.md`](METHOD.md#github-settings).
@@ -59,21 +116,37 @@ over the alerts that workflow uploads. Read back with the commands in
 
 ## 2. Scanners, Socket and gitleaks
 
-Every public repository, each checkout in a folder named after its repository (the runner refuses
-two checkouts with the same folder name):
+Every public repository, with stable distinct checkout names (the local `.github` checkout is
+`wirecat-github`; its GitHub identity is recorded separately):
+
+Use the variables from preflight. Start a new baseline with:
 
 ```sh
-RUN=~/Projects/AI/cli-private/security/runs/$(date +%F)
-P=~/Projects/AI
-C=($P/max-cli $P/tg-cli $P/cli-core $P/cli-base $P/cli-memo $P/drive-cli $P/zoom-cli $P/cli-docs
-  $P/cli-testing $P/community $P/.github)
-cli-testing security scan "$RUN" "${C[@]}" --previous <previous-run>
-cli-testing security socket "$RUN" "${C[@]}"
-for c in "${C[@]}" $P/cli-private; do
+node "$TESTING/dist/main.js" security scan "$RUN" "${C[@]}"
+node "$TESTING/dist/main.js" security socket "$RUN" "${C[@]}"
+for c in "${C[@]}"; do
   (cd "$c" && gitleaks git --no-banner --redact .; echo "$(basename "$c") gitleaks exit=$?")
 done
 ```
 
+For a complete approved previous baseline, add `--previous "$PREVIOUS"` to `security scan`.
+Run Socket only after a completed scanner stage and the owner's authentication/source-upload
+approval. If it is unavailable, record that stage as not run; do not call the METHOD audit complete.
+
+The private repository never goes through hosted source scanners. Respect the owner's allowed
+paths. Its CI-compatible secret check covers tracked Markdown/workflow history, excluding
+`secrets/`; it is narrower than the public full-history checks:
+
+```sh
+(cd "$PRIVATE" && gitleaks git --no-banner --redact \
+  --log-opts="--all -- *.md .github/workflows/*.yml :(exclude)secrets/**" .)
+```
+
+Record the private allowlist, excluded paths, exit code, hits and actual scanned commit count.
+Do not widen it to local casts, credentials or raw account artifacts. Verify the configured scan
+with a separate temporary synthetic repository and a known-positive fixture, and confirm a
+nonzero history before accepting a clean result. Do not put literal quotes inside `--log-opts`.
+
 A failed repository does not stop the others: `scan` names it on stderr and exits 1 after the last
 one. Rerun that repository into a new run folder once the cause is fixed. gitleaks runs from each
 repository's root, where it finds that repository's `.gitleaksignore`; record each exit code and
@@ -113,6 +186,9 @@ time.
 
 ## 7. Record
 
+Use the [regression checklist](REGRESSIONS.md) for positive and negative controls and the per-stage
+completion ledger. Include the accepted-risk input and exact source/release boundary.
+
 `REPORT.md` in the run folder: scanner results, verdict per checklist item with its basis, finding
 details, what this run did not cover, and the comparison with the previous run. Each finding into
 the private register. Fixes go in as pull requests, one per repository and theme.