diff --git a/CHANGELOG.md b/CHANGELOG.md index bd7fedd..b7d4e43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,9 @@ Every entry says what changed as a caller sees it, why, and what to watch for ### Added +- `security preflight` checks scanner pins, command availability, canonical repository paths and + aliases, GitHub analysis access and an optional comparison baseline without scanning source. + JSON diagnostics distinguish scanner readiness from the separately approved audit stages. - Drive AI extraction research drives the executable against a guarded loopback responder, checking evidence quotes, proposal persistence, provider failures and source changes. Historical npm and local candidate results are explicitly distinguished. diff --git a/README.md b/README.md index 24318b3..c641a44 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,8 @@ Install this package as a development dependency and build it before running fro Give every scan a new private output directory; a recorded stage is never overwritten. ```sh -cli-testing security scan ... +cli-testing security preflight ... [--previous ] +cli-testing security scan ... [--previous ] cli-testing security socket ... cli-testing compare security ``` diff --git a/docs/dev/ARCHITECTURE.md b/docs/dev/ARCHITECTURE.md index 4351fb1..577392e 100644 --- a/docs/dev/ARCHITECTURE.md +++ b/docs/dev/ARCHITECTURE.md @@ -11,6 +11,7 @@ Published as `@wirecat/cli-testing`, a development dependency of max-cli and tg- | [`src/leaks.ts`](../../src/leaks.ts) | bounded text-artifact and diagnostic leak checks | | [`src/contract-main.ts`](../../src/contract-main.ts) | the `cli-contract` executable | | [`src/runner.ts`](../../src/runner.ts) | pinned scanner stages and the explicitly live agent plan | +| [`src/preflight.ts`](../../src/preflight.ts) | read-only tooling, canonical repository and comparison-baseline checks | | [`src/runs.ts`](../../src/runs.ts) | private run layout, immutable stage records and legacy reads | | [`src/observations.ts`](../../src/observations.ts) | scanner candidates reduced to stable identifiers | | [`src/compare.ts`](../../src/compare.ts) | observation history and transcript redaction | diff --git a/docs/dev/BACKLOG.md b/docs/dev/BACKLOG.md index 6a8ce1d..10db5ac 100644 --- a/docs/dev/BACKLOG.md +++ b/docs/dev/BACKLOG.md @@ -19,8 +19,9 @@ the private `cli-private` repository, never here. ## Phase 1 — remaining security work -- **SEC-5 · P2** A second audit run of max-cli and tg-cli after the first audit's fixes ship, - compared with the first; the agent suite on tg-cli for the first time. +- **SEC-5 · P2** A complete new audit over the current repository set, with explicit scanner, + Socket, manual/false-positive and owner-approved agent stages. Start a fresh scanner baseline + if historical coverage is incomplete; compare only complete records with matching coverage. ## Phase 2 — contract diff --git a/docs/security/LESSONS.md b/docs/security/LESSONS.md index c4c649f..16c4d5d 100644 --- a/docs/security/LESSONS.md +++ b/docs/security/LESSONS.md @@ -35,6 +35,13 @@ Facts learned during the audits, with their source. Each cost time once. - **Trail of Bits `fp-check` installs a Stop hook on every session** that blocks stopping until the verification is visible in the conversation itself. Disabling the plugin takes effect from the next session; in the current one, turn it off in `/hooks`. +- **Zero scanned history is not a clean secret scan.** Gitleaks splits `--log-opts` itself; + embedded shell quotes can select zero commits. Keep the history/path expression in one shell + argument without internal quotes, verify a nonzero scanned count and use a separate synthetic + known-positive control. A private allowlisted scan does not establish whole-repository coverage. +- **A delivery closeout is not a new audit baseline.** Do not change historical incomplete runs + to pass comparison. Bootstrap a new scanner run without `--previous`, record its coverage and + track manual verification, Socket and live A7 independently. ## GitHub diff --git a/docs/security/METHOD.md b/docs/security/METHOD.md index d92d2c8..8508693 100644 --- a/docs/security/METHOD.md +++ b/docs/security/METHOD.md @@ -20,6 +20,11 @@ gh repo list WireCatLabs --limit 100 --json name,visibility --jq '.[] | "\(.name ## A run, step by step +Start with the read-only preflight in [RUNBOOK](RUNBOOK.md#preflight-and-baseline). Use a new +baseline without `--previous` when no complete comparable run exists. Historical gaps stay gaps; +scanner completion never substitutes for the manual/Socket/A7 stages. The +[regression checklist](REGRESSIONS.md) defines the evidence and stage ledger for a complete audit. + 1. `cli-testing security scan ... [--previous ]` creates the run folder, refuses tool versions other than the pins (or pins younger than two weeks), and calls [`scripts/security/scan`](../../scripts/security/scan). The script exports `origin/main` of each @@ -33,9 +38,10 @@ gh repo list WireCatLabs --limit 100 --json name,visibility --jq '.[] | "\(.name adds its exit code to `run.json`. The Socket CLI banner prints the start of the API token; the script passes `--no-banner` after the command (before it, the flag is not parsed). Raw output goes to `/raw//.json`, so two runs can be diffed. -3. gitleaks over the full history of every repository, `cli-private` included, from each - repository's root (it reads `.gitleaksignore` from the current folder): - `gitleaks git --no-banner --redact .`. Exit codes and hit counts go into `SCOPE.md`. +3. gitleaks over the full history of each public repository, from its root (it reads + `.gitleaksignore` from the current folder): `gitleaks git --no-banner --redact .`. The private + allowlist/exclusions must respect the owner's file permissions and are recorded separately; + see RUNBOOK. Record exit codes, hit counts and nonzero scanned history in `SCOPE.md`. 4. Read the GitHub settings that `scan` does not (commands in "GitHub settings") into `SCOPE.md`. 5. Do the manual review (section "Checklist"). Every item gets a verdict in `runs//REPORT.md`: safe, finding, or not applicable — each with the evidence that supports it. diff --git a/docs/security/REGRESSIONS.md b/docs/security/REGRESSIONS.md new file mode 100644 index 0000000..1497a2f --- /dev/null +++ b/docs/security/REGRESSIONS.md @@ -0,0 +1,56 @@ +# Audit regression checks + +Use this when preparing or repeating an audit after fixes. It helps reviewers prove that the +selected protection works, ordinary supported workflows still work, and incomplete stages remain +visible. Read the owner's accepted-risk decisions before selecting controls; this checklist does +not authorize account access, settings changes, publication or live messenger actions. + +## Record the boundary + +For each selected fix, record privately: finding identifier, source commit, released package and +exact dependency pins, owning test, executed command, positive and negative results, limitations +and reviewer. Test current source and the installed release separately when they differ. A test +name or green unrelated CI run is not proof that the changed behavior was exercised. + +Use synthetic accounts/data and temporary state for process tests. Never open the owner's store, +read credential files, install a real user timer or send a messenger message as a routine PoC. + +## Protection and compatibility controls + +| Area | Protection control | Allowed-workflow control | +| --- | --- | --- | +| Source-analysis gate | A synthetic critical SARIF result blocks; missing/unresolvable rule metadata fails | A lower-severity result is reported under the approved policy; resolve rules in driver and extension components | +| Scheduled authority | Under the inherited read-only lock, send and authority-granting config/permissions/recipients/schedule changes refuse before execution | Allowed reads and reply-text preparation work; do not claim the CLI lock sandboxes other programs | +| Credential paths | Synthetic canonical, symlinked and platform-case variants of protected paths refuse before bytes are read or transferred | Ordinary hidden working folders remain allowed; exercise only the explicitly supported override | +| Parser complexity | A bounded adversarial fixture completes inside a measured resource budget or fails with the defined error | Ordinary valid HTML/documents preserve expected text; accepted long-document behavior stays supported | +| MCP text boundaries | Incoming text is framed as data and hidden controls are visible where the released contract promises it | Original Unicode writes and ordinary machine JSON follow the accepted compatibility policy | +| File creation | A pre-existing target/symlink refuses an exclusive temporary/output write and retains sentinel contents | A new target succeeds; normal valid output is not truncated or overwritten | +| Deployment/publishing | Actual secret-name metadata and environment branch rules match the intended credential boundary | The authorized main deployment/publisher succeeds with those scoped credentials; workflow text alone is insufficient | +| Audit tooling | Wrong pins, missing source analysis, failed scanners and incomplete/mismatched comparison coverage remain failures or explicit gaps | Canonical symlinks work, supported scanner exit codes retain observations, and a new baseline needs no invented previous run | + +For secret scanners, use a separate temporary synthetic Git history with a known-positive fixture. +Require a detection, then verify that the intended clean control scans nonzero commits. Record the +actual allowlist and exclusions. Do not copy canaries into maintained source or publish findings. + +## Completion ledger + +Keep this table in the private run report, with a source/artifact link for every status. Use +`run`, `not run`, `failed`, or `not applicable` with a reason. A successful CLI `scan` means only +that scanner stage completed; it does not fill the other rows. + +| Stage | Required evidence | +| --- | --- | +| Preflight | Tooling commit, paths/aliases, tool pins, source and GitHub access, baseline choice | +| Static scanners | Source commits, versions, exit codes, coverage/ignore inventory and retained parser errors | +| CodeQL | Analysis availability, open-alert snapshot and manual triage; 404 is not zero alerts | +| Secrets | Public full-history results; private allowed history scope; positive control and nonzero scanned history | +| Socket | Approved source-upload scope, pinned CLI, same commits, valid successful report; owner-terminal authentication | +| Manual review | Assigned S1–S10/A1–A7 owners, accepted-risk input, source/release boundary and evidence per verdict | +| False-positive verification | Bounded PoC and negative control, reviewer verdict and limitations before dismissal | +| Agent A7 | Explicit approval for cast/payloads, host-refused canary, utility checks, attempted writes and verified cleanup | +| Comparison | Complete comparable scanner records with matching repository/scanner coverage; missing coverage cannot become fixed observations | +| Delivery follow-up | Fix PR, exact published consumer/dependency versions, targeted regression evidence and updated private register | + +An audit is complete only when every required row has evidence, or an explicit approved +not-applicable decision. `not run` and `failed` rows keep the audit partial. Keep historical +reports unchanged; record later delivery in a linked closeout and the current findings register. diff --git a/docs/security/RUNBOOK.md b/docs/security/RUNBOOK.md index f5105d8..2bb67e9 100644 --- a/docs/security/RUNBOOK.md +++ b/docs/security/RUNBOOK.md @@ -1,6 +1,7 @@ # Security audit — runbook -The commands of one audit run, in order. Why each step exists: [`METHOD.md`](METHOD.md). The +Use this before starting an audit to prepare the checkouts, validate the tools and record each +stage without overstating coverage. Why each step exists: [`METHOD.md`](METHOD.md). The first run was 2026-10-09; its results are private (`cli-private/security/runs/2026-10-09/`). ## 0. Tools, pinned @@ -9,8 +10,11 @@ A version at least two weeks old on the day of the run, recorded in `SCOPE.md` b `cli-testing security scan` refuses any other version. ```sh -uv tool install semgrep==1.178.0 -uv tool install zizmor==1.30.1 +AUDIT_TOOLS="${XDG_CACHE_HOME:-$HOME/.cache}/wirecat-audit/semgrep-1.178.0" +test -x "$AUDIT_TOOLS/bin/python" || uv venv "$AUDIT_TOOLS" +uv pip install --python "$AUDIT_TOOLS/bin/python" semgrep==1.178.0 zizmor==1.30.1 +export PATH="$AUDIT_TOOLS/bin:$PATH" +export SEMGREP_SEND_METRICS=off SEMGREP_ENABLE_VERSION_CHECK=0 # gitleaks: the release binary, checked against its checksums file the same way as osv-scanner # osv-scanner: the release binary, checked against the release's own SHA256SUMS V=2.6.0; D=$(mktemp -d); cd "$D" @@ -18,6 +22,9 @@ gh release download v$V -R google/osv-scanner -p osv-scanner_linux_amd64 -p osv- grep ' osv-scanner_linux_amd64$' osv-scanner_SHA256SUMS | sha256sum -c - && install -m 755 osv-scanner_linux_amd64 ~/.local/bin/osv-scanner ``` +Keep the audit environment first on PATH for the whole run: Semgrep also starts `osemgrep` +through PATH. This preserves any different global installation. + Socket CLI runs through `pnpm dlx @socketsecurity/cli@1.1.179` inside the script. Log in once, **in your own terminal**, never through an agent: `pnpm dlx @socketsecurity/cli@1.1.179 login`. @@ -26,7 +33,57 @@ Claude Code plugins: `claude plugin marketplace add trailofbits/skills`, then in fp-check agentic-actions-auditor differential-review` (`claude plugin install @trailofbits`). Disable `fp-check` again after the run — its Stop hook fires on every session. -## 1. GitHub settings — apply, then read back +## Preflight and baseline + +Run from a clean cli-testing worktree built from the commit you intend to use. This does not +require a globally installed executable: + +```sh +P="$HOME/Projects/AI" +TESTING="$P/cli-testing" +PRIVATE="$P/max-cli/docs_ai" +RUN="$PRIVATE/security/runs/$(date +%F)" +C=("$P/max-cli" "$P/tg-cli" "$P/cli-core" "$P/cli-base" "$P/cli-memo" "$P/drive-cli" "$P/zoom-cli" "$P/cli-docs" + "$P/cli-testing" "$P/community" "$P/wirecat-github") +cd "$TESTING" +pnpm install --frozen-lockfile +pnpm build +node "$TESTING/dist/main.js" security preflight "${C[@]}" +``` + +Set `TESTING` to the clean tooling worktree and `PRIVATE` to the private checkout in use. Verify +that the output directory is inside that private checkout. The public source repositories are +exported from `origin/main`; unrelated local edits are not scanned. Fetch the intended refs before +preflight. Record both source commits and the cli-testing commit in the new scope. + +A local `cli-core` symlink to `cli-messaging` is supported: keep the canonical `cli-core` name in +`C`, and do not include both aliases. Preflight rejects duplicate physical checkouts, unrecognized GitHub +origins, private source repositories, unavailable commands, +unsupported scanner pins, unreadable CodeQL analysis and invalid comparison metadata. It only +reads Git/GitHub metadata and tool versions; it does not scan source, create a run, read credential +files, log in, change settings or authorize live tests. Socket authentication remains a separate +owner-terminal check; preflight does not start Socket or fetch a token. + +For the first valid scanner baseline, omit `--previous`. An older report may still be used as +manual historical evidence; never change its `complete` flag or manufacture missing outputs. +For a later comparison, set an existing complete baseline explicitly: + +```sh +PREVIOUS=/absolute/path/to/a/completed-private-run +node "$TESTING/dist/main.js" security preflight "${C[@]}" --previous "$PREVIOUS" +``` + +The baseline needs a complete security `run.json` and recorded coverage. `compare` additionally +requires identical repository/scanner coverage. A missing analysis, failed parser or absent stage +is a gap, not a fixed finding. A CodeQL 404 may mean absent analysis or insufficient access; resolve +it or record the repository as incomplete. Never replace it with an empty alerts array. A +successful scanner stage is not a complete METHOD audit: track manual review, false-positive +verification, Socket, full-history secret scanning and A7 separately in the report. + +## 1. GitHub settings — read back, then apply approved changes + +First read settings with the commands in METHOD. An audit does not authorize settings changes. +Apply the following only when the owner has approved the specific correction, then read it back. New public repositories get their settings from the organisation configuration `WireCat public`; check that it is still the default with the command in [`METHOD.md`](METHOD.md#github-settings). @@ -59,21 +116,37 @@ over the alerts that workflow uploads. Read back with the commands in ## 2. Scanners, Socket and gitleaks -Every public repository, each checkout in a folder named after its repository (the runner refuses -two checkouts with the same folder name): +Every public repository, with stable distinct checkout names (the local `.github` checkout is +`wirecat-github`; its GitHub identity is recorded separately): + +Use the variables from preflight. Start a new baseline with: ```sh -RUN=~/Projects/AI/cli-private/security/runs/$(date +%F) -P=~/Projects/AI -C=($P/max-cli $P/tg-cli $P/cli-core $P/cli-base $P/cli-memo $P/drive-cli $P/zoom-cli $P/cli-docs - $P/cli-testing $P/community $P/.github) -cli-testing security scan "$RUN" "${C[@]}" --previous -cli-testing security socket "$RUN" "${C[@]}" -for c in "${C[@]}" $P/cli-private; do +node "$TESTING/dist/main.js" security scan "$RUN" "${C[@]}" +node "$TESTING/dist/main.js" security socket "$RUN" "${C[@]}" +for c in "${C[@]}"; do (cd "$c" && gitleaks git --no-banner --redact .; echo "$(basename "$c") gitleaks exit=$?") done ``` +For a complete approved previous baseline, add `--previous "$PREVIOUS"` to `security scan`. +Run Socket only after a completed scanner stage and the owner's authentication/source-upload +approval. If it is unavailable, record that stage as not run; do not call the METHOD audit complete. + +The private repository never goes through hosted source scanners. Respect the owner's allowed +paths. Its CI-compatible secret check covers tracked Markdown/workflow history, excluding +`secrets/`; it is narrower than the public full-history checks: + +```sh +(cd "$PRIVATE" && gitleaks git --no-banner --redact \ + --log-opts="--all -- *.md .github/workflows/*.yml :(exclude)secrets/**" .) +``` + +Record the private allowlist, excluded paths, exit code, hits and actual scanned commit count. +Do not widen it to local casts, credentials or raw account artifacts. Verify the configured scan +with a separate temporary synthetic repository and a known-positive fixture, and confirm a +nonzero history before accepting a clean result. Do not put literal quotes inside `--log-opts`. + A failed repository does not stop the others: `scan` names it on stderr and exits 1 after the last one. Rerun that repository into a new run folder once the cause is fixed. gitleaks runs from each repository's root, where it finds that repository's `.gitleaksignore`; record each exit code and @@ -113,6 +186,9 @@ time. ## 7. Record +Use the [regression checklist](REGRESSIONS.md) for positive and negative controls and the per-stage +completion ledger. Include the accepted-risk input and exact source/release boundary. + `REPORT.md` in the run folder: scanner results, verdict per checklist item with its basis, finding details, what this run did not cover, and the comparison with the previous run. Each finding into the private register. Fixes go in as pull requests, one per repository and theme. diff --git a/docs/security/review-prompts.md b/docs/security/review-prompts.md index d4540e9..b1a54d2 100644 --- a/docs/security/review-prompts.md +++ b/docs/security/review-prompts.md @@ -1,6 +1,6 @@ # Manual review — reviewer prompts -The six read-only reviewers of a run (step 4 of [`RUNBOOK.md`](RUNBOOK.md)), one per area, run in +The read-only reviewers of a run (step 4 of [`RUNBOOK.md`](RUNBOOK.md)), one per area, run in parallel. Each gets the same preamble, its own checklist items from [`METHOD.md`](METHOD.md), and the same report shape. @@ -11,7 +11,12 @@ the same report shape. > export of `origin/main` at `/{}`. These are Node/TypeScript CLIs driving the > owner's real messenger accounts; shared code lives in the libraries they depend on. Report ONLY > real candidate issues plus a one-line "safe because …" per area. Skip tests, `scripts/`, `bench/` -> unless shipped. +> unless shipped or involved in installation, release, deployment or audit tooling. Never read +> credential/config/session files or private casts. Inputs also include the owner-approved risk +> decisions, the prior findings/closeout and the exact released dependency versions. Preserve +> accepted compatibility choices; report a new regression separately from an accepted risk. +> Distinguish current source, released behavior and executed evidence. Do not infer that a +> setting or credential scope is correct merely because a workflow names an environment. ## Report shape, for every reviewer @@ -22,7 +27,9 @@ the same report shape. > "inferred". > 3. What you could not check. -## The six areas + + +## The review areas 1. **Install, update, processes — S1, S2, S3.** The install script and what it imports; the self-update path (can a registry answer reach a shell or a path?); every process spawn — shell or @@ -42,3 +49,15 @@ the same report shape. libraries come from and whether their hashes are checked. 6. **CodeQL triage.** Every open alert: real · not reachable · not shipped · false positive · fixed, with a dismissal reason for each non-real one. +7. **CI, release and deployment — S9.** Every repository's jobs and reusable workflows, dependency + execution beside credentials, action pins, permissions, cache trust, production environment + branch policy and repository versus environment secret metadata. Read back settings only; + no secret values, policy edits, publication or test dispatch without task authorization. +8. **Third-party dependencies — S10.** Production versus development reachability, package + contents, install scripts and scanner coverage. Resolve the exact published dependency tree; + distinguish an advisory from a demonstrated reachable defect. Track any audit ignore's reason + and expiry without expanding it to hide an unresolved failure. + +Each checklist item S1–S10 and A1–A7 needs an assigned owner and an evidence-backed verdict. +Use [REGRESSIONS.md](REGRESSIONS.md) to retest both refusals and allowed workflows. Unassigned, +unavailable and unexecuted items remain explicit gaps; a reviewer cannot mark an inference run. diff --git a/scripts/smoke.ts b/scripts/smoke.ts index 12eb9a0..30e19d0 100644 --- a/scripts/smoke.ts +++ b/scripts/smoke.ts @@ -13,6 +13,7 @@ import { runContracts, SUITE_GROUPS, scanLeaks, + securityPreflight, } from "../src/index.ts" assert.ok(SUITE_GROUPS.includes("security")) @@ -23,4 +24,5 @@ assert.equal(contractToolSnapshot([]), "[]\n") assert.equal(typeof runContracts, "function") assert.deepEqual(findLeaks("ordinary diagnostic"), []) assert.equal(typeof scanLeaks, "function") +assert.equal(typeof securityPreflight, "function") console.log("smoke ok") diff --git a/src/index.ts b/src/index.ts index 7b7ea4b..d42be83 100644 --- a/src/index.ts +++ b/src/index.ts @@ -12,5 +12,6 @@ export { runContracts, } from "./contracts.js" export { findLeaks, type LeakFinding, scanLeaks } from "./leaks.js" +export { type PreflightCheck, securityPreflight } from "./preflight.js" export { type ProcessRunner, runAgent, runSecurity, TOOL_PINS } from "./runner.js" export { readRun } from "./runs.js" diff --git a/src/main.ts b/src/main.ts index 5cc1f9d..ad0625a 100644 --- a/src/main.ts +++ b/src/main.ts @@ -5,10 +5,12 @@ import { fileURLToPath } from "node:url" import { parseArgs } from "node:util" import { compareRuns } from "./compare.js" import { scanLeaks } from "./leaks.js" +import { securityPreflight } from "./preflight.js" import { type ProcessRunner, runAgent, runSecurity } from "./runner.js" import { readRun } from "./runs.js" -export const HELP = `cli-testing security scan ... [--previous ] +export const HELP = `cli-testing security preflight ... [--previous ] +cli-testing security scan ... [--previous ] cli-testing security socket ... cli-testing security no-leak cli-testing agent --live [--previous ] @@ -24,6 +26,13 @@ export const runCli = (argv: string[], deps: { run?: ProcessRunner; now?: () => options: { help: { type: "boolean", short: "h" }, live: { type: "boolean" }, previous: { type: "string" } }, }) if (values.help || args.length === 0) return { exit: 0, output: HELP } + if (args[0] === "security" && args[1] === "preflight") { + const report = securityPreflight(args.slice(2), { + ...deps, + ...(values.previous === undefined ? {} : { previous: values.previous }), + }) + return { exit: report.ready ? 0 : 1, output: `${JSON.stringify(report)}\n` } + } if (args[0] === "security" && args[1] === "no-leak" && args.length === 4) { try { const policy = JSON.parse(readFileSync(resolve(args[3] as string), "utf8")) as { diff --git a/src/preflight.test.ts b/src/preflight.test.ts new file mode 100644 index 0000000..06309a9 --- /dev/null +++ b/src/preflight.test.ts @@ -0,0 +1,104 @@ +import { mkdirSync, mkdtempSync, readFileSync, symlinkSync, writeFileSync } from "node:fs" +import { tmpdir } from "node:os" +import { basename, join } from "node:path" +import { describe, expect, it } from "vitest" +import { runCli } from "./main.js" +import { securityPreflight } from "./preflight.js" +import { type ProcessRunner, TOOL_PINS } from "./runner.js" + +const now = () => new Date("2026-10-16T00:00:00Z") +const fixture = () => mkdtempSync(join(tmpdir(), "preflight-")) +const backend = + (calls: string[], { codeql = true, visibility = "public" } = {}): ProcessRunner => + (command, args) => { + calls.push(`${command} ${args.join(" ")}`) + if (command === "git" && args.includes("get-url")) { + const name = basename(args[1] as string) + return { exit: 0, stdout: `git@github.com:fixture/${name === "cli-messaging" ? "cli-core" : name}.git\n` } + } + if (command === "git" && args.includes("rev-parse")) return { exit: 0, stdout: `${"a".repeat(40)}\n` } + if (command === "gh" && args[0] === "api") + return args.includes("type") + ? { exit: codeql ? 0 : 1, stdout: codeql ? "array\n" : "" } + : { exit: 0, stdout: `${visibility}\n` } + if (args.length === 1 && (args[0] === "--version" || args[0] === "version")) + return { exit: 0, stdout: `${TOOL_PINS[command as keyof typeof TOOL_PINS]?.version ?? "synthetic-version"}\n` } + throw new Error("Preflight attempted a non-preflight operation") + } + +describe("security preflight", () => { + it("accepts the canonical symlink without scanning source or creating run artifacts", () => { + const root = fixture(), + target = join(root, "cli-messaging"), + alias = join(root, "cli-core"), + calls: string[] = [] + mkdirSync(target) + symlinkSync(target, alias) + const result = runCli(["security", "preflight", alias], { run: backend(calls), now }) + expect(result.exit).toBe(0) + const report = JSON.parse(result.output) + expect(report.ready).toBe(true) + expect(report.baseline).toBe("new") + expect(report.manualStages).toHaveLength(3) + expect(calls.some((call) => call.includes("security/scan") || call.startsWith("socket "))).toBe(false) + const duplicate = securityPreflight([alias, target], { run: backend([]), now }) + expect(duplicate.ready).toBe(false) + expect(duplicate.checks).toContainEqual(expect.objectContaining({ name: "cli-messaging identity", status: "fail" })) + }) + + it("requires a complete recorded comparison baseline without changing historical records", () => { + const root = fixture(), + repo = join(root, "public-repo"), + previous = join(root, "previous") + mkdirSync(repo) + mkdirSync(previous) + const path = join(previous, "run.json") + const record = { + schema: 1, + suite: "security", + complete: true, + observations: [], + known: [], + coverage: ['["public-repo","semgrep"]'], + } + for (const value of [ + "invalid JSON", + JSON.stringify({ ...record, complete: false }), + JSON.stringify({ ...record, coverage: [] }), + JSON.stringify({ ...record, suite: "agent" }), + ]) { + writeFileSync(path, value) + expect(securityPreflight([repo], { run: backend([]), now, previous }).ready).toBe(false) + expect(readFileSync(path, "utf8")).toBe(value) + } + writeFileSync(path, JSON.stringify(record)) + const ready = securityPreflight([repo], { run: backend([]), now, previous }) + expect(ready.ready).toBe(true) + expect(ready.baseline).toBe("comparison") + expect(securityPreflight([repo], { run: backend([]), now, previous: join(root, "missing") }).ready).toBe(false) + }) + + it("reports inaccessible CodeQL and private sources before any scanner runs", () => { + const root = fixture(), + repo = join(root, "public-repo") + mkdirSync(repo) + const gap = securityPreflight([repo], { run: backend([], { codeql: false }), now }) + expect(gap.ready).toBe(false) + expect(gap.checks).toContainEqual(expect.objectContaining({ name: "public-repo CodeQL", status: "fail" })) + const calls: string[] = [] + const privateSource = securityPreflight([repo], { run: backend(calls, { visibility: "private" }), now }) + expect(privateSource.ready).toBe(false) + expect(calls.some((call) => call.includes("code-scanning"))).toBe(false) + }) + + it("reports missing commands, unsupported pins, young pins and absent checkouts together", () => { + const root = fixture() + const result = securityPreflight([join(root, "missing")], { + run: () => ({ exit: 2, stdout: "unavailable" }), + now: () => new Date("2026-09-01"), + }) + expect(result.ready).toBe(false) + expect(result.checks.filter((check) => check.status === "fail").length).toBeGreaterThan(10) + expect(securityPreflight([], { run: backend([]), now }).ready).toBe(false) + }) +}) diff --git a/src/preflight.ts b/src/preflight.ts new file mode 100644 index 0000000..bef10ba --- /dev/null +++ b/src/preflight.ts @@ -0,0 +1,115 @@ +import { existsSync, realpathSync } from "node:fs" +import { basename, join, resolve } from "node:path" +import { type ProcessRunner, runProcess, TOOL_PINS } from "./runner.js" +import { readRun } from "./runs.js" + +export interface PreflightCheck { + name: string + status: "pass" | "fail" + detail: string +} + +export const securityPreflight = ( + repositories: string[], + { + run = runProcess, + now = () => new Date(), + previous, + }: { run?: ProcessRunner; now?: () => Date; previous?: string } = {}, +) => { + const checks: PreflightCheck[] = [] + const add = (name: string, passed: boolean, detail: string) => + checks.push({ name, status: passed ? "pass" : "fail", detail }) + for (const [name, args] of [ + ["git", ["--version"]], + ["bash", ["--version"]], + ["node", ["--version"]], + ["pnpm", ["--version"]], + ["gh", ["--version"]], + ["jq", ["--version"]], + ["gitleaks", ["version"]], + ] as const) + add(name, run(name, [...args]).exit === 0, "command must be available; record its version in SCOPE.md") + + for (const [name, pin] of Object.entries(TOOL_PINS)) { + const oldEnough = now().getTime() - Date.parse(pin.released) >= 14 * 86_400_000 + add(`${name} pin age`, oldEnough, `${pin.version} must be at least two weeks old`) + if (name === "socket") continue + const version = run(name, ["--version"]) + add(name, version.exit === 0 && version.stdout.split(/\s+/).includes(pin.version), `requires ${pin.version}`) + } + + add("repositories", repositories.length > 0, "supply the public repository checkouts explicitly") + const physical = new Set() + const names = new Set() + for (const repository of repositories) { + const path = resolve(repository) + const name = basename(path) + let target: string + try { + target = realpathSync(path) + } catch { + add(path, false, "checkout directory does not exist") + continue + } + const distinct = !physical.has(target) && !names.has(name) + physical.add(target) + names.add(name) + add(`${name} identity`, distinct, "scan each physical checkout once, with distinct folder names") + const remote = run("git", ["-C", path, "remote", "get-url", "origin"]) + const slug = remote.stdout + .trim() + .match(/^(?:git@github\.com:|https:\/\/github\.com\/)([\w.-]+\/[\w.-]+?)(?:\.git)?$/)?.[1] + const head = run("git", ["-C", path, "rev-parse", "--verify", "origin/main^{commit}"]) + add(`${name} main`, head.exit === 0 && /^[0-9a-f]{40}\s*$/.test(head.stdout), "fetch origin/main before the scan") + const identified = remote.exit === 0 && slug !== undefined + add(`${name} origin`, identified, `${slug ?? "unrecognized origin"}; use stable distinct checkout names`) + if (!identified || slug === undefined) continue + const metadata = run("gh", ["api", `repos/${slug}`, "--jq", ".visibility"]) + add( + `${name} access`, + metadata.exit === 0 && metadata.stdout.trim() === "public", + "only public source repositories go to scanners", + ) + if (metadata.exit !== 0 || metadata.stdout.trim() !== "public") continue + const codeql = run("gh", ["api", `repos/${slug}/code-scanning/alerts?per_page=1`, "--jq", "type"]) + add( + `${name} CodeQL`, + codeql.exit === 0 && codeql.stdout.trim() === "array", + "unavailable analysis is a coverage gap, never zero findings", + ) + } + + if (previous !== undefined) { + let valid = false + try { + if (existsSync(join(previous, "run.json"))) { + const record = readRun(previous) + valid = + record.suite === "security" && + record.complete && + Array.isArray(record.coverage) && + record.coverage.length > 0 && + record.coverage.every((item) => typeof item === "string") + } + } catch { + valid = false + } + add( + "previous run", + valid, + "requires a complete security run.json with recorded coverage; otherwise start a new baseline without --previous", + ) + } + + return { + ready: checks.every((check) => check.status === "pass"), + baseline: previous === undefined ? "new" : "comparison", + checks, + manualStages: [ + "Socket uses the pinned CLI through pnpm; verify authentication in the owner's terminal before source upload", + "Record manual review, false-positive verification, full-history secret-scan coverage and limitations separately", + "Live A7 requires owner-approved accounts, chats and payloads; preflight grants no live authorization", + ], + } +} diff --git a/tsconfig.json b/tsconfig.json index d2a8bac..0e0d60e 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -1,5 +1,5 @@ { - "extends": "@wirecat/cli-base/tsconfig.base.json", + "extends": "./node_modules/@wirecat/cli-base/config/tsconfig.base.json", "compilerOptions": { "rootDir": "src", "outDir": "dist" diff --git a/vitest.config.ts b/vitest.config.ts index 42fc4c3..23247c8 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -1,6 +1,9 @@ +import { fileURLToPath } from "node:url" import { defineConfig } from "vitest/config" export default defineConfig({ + // The build config excludes test/sandbox.ts; automatic discovery climbs into the parent checkout. + tsconfig: fileURLToPath(new URL("./tsconfig.json", import.meta.url)), test: { // Agents run suites side by side on 24 cores; one worker per core ran the machine out of memory (2026-10-11). maxWorkers: 4,