test(drive): validate AI evidence extraction through the executable

CHANGELOG.md

@@ -17,6 +17,9 @@ Every entry says what changed as a caller sees it, why, and what to watch for 
 
 ### Added
 
+- Drive AI extraction research drives the executable against a guarded loopback responder,
+  checking evidence quotes, proposal persistence, provider failures and source changes.
+  Historical npm and local candidate results are explicitly distinguished.
 - Complex Drive research acceptance: compiled-package SDK fixtures exercise nested content,
   interrupted exports and source lifecycle with a real temporary store; live multi-tab comparisons
   publish aggregate counts only, with private source details kept separately. Exact 0.11.2 and

performance/documents/README.md

@@ -2,6 +2,9 @@
 
 Measurements of how documents become text before they are indexed and searched.
 
+- [drive-ai](drive-ai/README.md): executable extraction checks with a loopback model responder,
+  source evidence, explicit saves, error handling and an in-flight source change.
+
 - [drive-complex](drive-complex/README.md): seven read-only multi-tab Google Docs plus compiled
   SDK-fixture checks for nested content, edit/move/trash recovery, scope exit/re-entry and context retention.
 

performance/documents/drive-ai/README.md

@@ -0,0 +1,70 @@
+# Drive AI extraction
+
+Drive requires an exact source quote for every extracted fact or memory. Its shared model
+gateway appended a blanket instruction not to quote source data. This check reproduces that
+contradiction and validates an explicit source-quote opt-in before publishing it.
+
+## Setup
+
+2026-10-15, Linux, Node 24.19.0. The baseline is npm Drive **0.12.0**, Core **0.289.0**,
+Base **0.33.0**. The local candidate is compiled Drive 0.12.0 plus the extraction caller change
+and Core PR [#1317](https://github.com/WireCatLabs/cli-core/pull/1317), with Base **0.35.0**.
+Its Core package metadata still says 0.292.0; it is **not the npm 0.292.0 artifact**. Candidate
+results record compiled module hashes to distinguish it. No fixed npm release is claimed here.
+
+[run.mjs](run.mjs) starts the actual compiled Drive executable in child processes. It seeds
+synthetic documents through the public shared-store API, then uses ordinary CLI commands,
+the real model gateway, an OpenAI-compatible loopback HTTP responder and the real SQLite store.
+Only the in-flight source-change case calls the compiled public command entry point in the
+parent, supplying a synthetic Google source to perform a real sync while the model response waits.
+
+```sh
+D=$(mktemp -d -p /var/tmp) && T=$(mktemp -d -p /var/tmp) && HOME=$T MESSAGING_STORE=$T/store WIRECAT_STATE_DIR=$T/state NPM_CONFIG_USERCONFIG=/dev/null npm install --prefix "$D" --ignore-scripts --no-audit --no-fund @wirecat/drive-cli@0.12.0
+T=$(mktemp -d -p /var/tmp) && HOME=$T MESSAGING_STORE=$T/store WIRECAT_STATE_DIR=$T/state NPM_CONFIG_USERCONFIG=/dev/null node performance/documents/drive-ai/run.mjs "$D/node_modules/@wirecat/drive-cli" --baseline
+```
+
+For a locally built candidate, substitute its package directory and `--candidate`. This mode
+records hashes and labels the result as local, rather than npm. Temporary state remains for
+inspection. Each run refuses separate physical Base instances in Drive and Core.
+
+[guard.mjs](guard.mjs) permits child fetch requests only to the exact loopback origin and
+chat-completion path; redirects, other HTTP requests, process launches and keyring access are
+refused. Child environments contain only the runner's temporary paths and an allowlist of
+process settings. The responder asserts that no Authorization header is present. No Google
+account, owner database, real document or model credential is used.
+
+## Results
+
+Both the [baseline](results-baseline.json) and [candidate](results-candidate.json) passed
+**59 CLI command checks**, with **12 loopback provider requests**, across seven behavior groups.
+The baseline explicitly expects the contradictory quote policy; passing that baseline does not
+mean its prompt is correct. The candidate requires the explicit evidence-quote policy and the
+absence of the blanket prohibition.
+
+| Behavior | Observed result |
+| --- | --- |
+| Preview | No writes; source hash, reference, provider and model retained |
+| Select one document tab | Sibling tab content excluded from the outbound request |
+| Explicit save and review | One scoped fact and one scoped memory saved as proposals with source evidence; explicit confirmation succeeds |
+| Invalid or failed response | Invented quote, invalid kind, oversized batch, malformed JSON, truncated response, HTTP 429 and HTTP 500 all save nothing; provider error body is absent from output |
+| Empty extraction | Successful no-op with no new proposals |
+| Guards before network | Offline, read-only, policy denial, absent tab, disabled provider and dry run do not contact the responder |
+| Source changes during extraction | Actual sync changes source hash; extraction refuses stale proposals with `document.changed`; the earlier confirmed memory becomes stale |
+
+Separately, the Drive candidate's **195 unit tests passed**, with **90.28% line coverage**.
+The shared Core baseline passed **4,567 tests**, with four existing skips and **95.11% line
+coverage**, before the concurrent meeting API/Base 0.35 changes were rebased in. The rebased
+Core candidate passed its **29 focused model tests** and typecheck on Base 0.35. These scopes
+are separate; the full earlier Core run does not certify the later unrelated changes.
+
+## Decision and limits
+
+Expose `ModelRequest.allowSourceQuotes: true` in Core and opt in only from Drive's evidence
+extraction caller. Leave ordinary requests' existing no-copy instruction intact. Source data
+remains in a separate message that instructs the model to treat it as untrusted content.
+
+The deterministic responder establishes transport, policy selection, validation, persistence
+and source-change handling. It does not establish real model extraction quality, resistance
+to prompt injection or compliance with instructions. No live AI-provider call or Google write
+was made. Publishing and validating a fresh installed npm artifact remain separate follow-up
+steps; candidate results must not be presented as released-package evidence.

performance/documents/drive-ai/guard.mjs

@@ -0,0 +1,37 @@
+import assert from "node:assert/strict"
+import { createRequire, syncBuiltinESMExports } from "node:module"
+
+const origin = new URL(process.env.DRIVE_TEST_MODEL_ORIGIN).origin
+assert.equal(new URL(origin).hostname, "127.0.0.1")
+const require = createRequire(`${process.env.DRIVE_TEST_PACKAGE}/package.json`)
+const deny = () => {
+  throw new Error("Unexpected network/browser/keyring access")
+}
+const originalFetch = globalThis.fetch
+globalThis.fetch = (input, init) => {
+  const url = new URL(input instanceof Request ? input.url : String(input))
+  if (url.origin !== origin || url.pathname !== "/v1/chat/completions" || init?.method !== "POST") deny()
+  return originalFetch(input, { ...init, redirect: "error" })
+}
+for (const name of ["node:http", "node:https"]) {
+  const module = require(name)
+  module.request = deny
+  module.get = deny
+}
+for (const name of ["exec", "execFile", "spawn", "fork", "execSync", "execFileSync", "spawnSync"])
+  require("node:child_process")[name] = deny
+syncBuiltinESMExports()
+const baseRequire = createRequire(require.resolve("@wirecat/cli-base"))
+const keyring = baseRequire.resolve("@napi-rs/keyring")
+require.cache[keyring] = {
+  id: keyring,
+  filename: keyring,
+  loaded: true,
+  exports: {
+    Entry: class {
+      constructor() {
+        deny()
+      }
+    },
+  },
+}

performance/documents/drive-ai/results-baseline.json

@@ -0,0 +1,19 @@
+{
+  "passed": true,
+  "data": "synthetic",
+  "real_model": false,
+  "drive": "0.12.0",
+  "core": "0.289.0",
+  "commands": 59,
+  "provider_requests": 12,
+  "quote_policy": "known-conflict-baseline",
+  "checks": [
+    "preview-does-not-persist-and-source-hash-is-retained",
+    "tab-selection-does-not-send-sibling-text",
+    "explicit-save-creates-reviewable-scoped-facts-and-memories",
+    "invalid-or-failed-provider-batches-save-nothing-and-leak-no-response-body",
+    "empty-proposal-batch-is-a-noop",
+    "offline-readonly-denied-policy-missing-tab-disabled-provider-and-dry-run-precede-network",
+    "real-sync-during-model-request-refuses-stale-proposals"
+  ]
+}

performance/documents/drive-ai/results-candidate.json

@@ -0,0 +1,24 @@
+{
+  "passed": true,
+  "data": "synthetic",
+  "real_model": false,
+  "drive": "0.12.0",
+  "core": "0.292.0",
+  "build": "local-candidate",
+  "code_hashes": {
+    "drive_extraction": "b39ace7774010e616deff2e5e985507f0cd1cef179cfda8ba90ada85efb77c2b",
+    "core_model_types": "8d058371f06f83d95bdee2656c5fb9f6b3520a7f2f2e6af782e2fb8163dc4cca"
+  },
+  "commands": 59,
+  "provider_requests": 12,
+  "quote_policy": "explicit-evidence-quotes",
+  "checks": [
+    "preview-does-not-persist-and-source-hash-is-retained",
+    "tab-selection-does-not-send-sibling-text",
+    "explicit-save-creates-reviewable-scoped-facts-and-memories",
+    "invalid-or-failed-provider-batches-save-nothing-and-leak-no-response-body",
+    "empty-proposal-batch-is-a-noop",
+    "offline-readonly-denied-policy-missing-tab-disabled-provider-and-dry-run-precede-network",
+    "real-sync-during-model-request-refuses-stale-proposals"
+  ]
+}