From 3fa60d69af92c67301db43f25f7c848489d6bea2 Mon Sep 17 00:00:00 2001 From: debuggingfuture Date: Fri, 9 Oct 2026 00:54:16 +0800 Subject: [PATCH 1/3] fix(ci): partition gate tests and report base listing timeouts --- .../rulesets/contextful-gate.proposed.json | 7 +- AGENTS.md | 20 +- spec/81-engineering.md | 10 +- spec/adr/A-assurance.md | 6 +- spec/cards/assurance.md | 2 +- spec/spec.lock.json | 251 ++++++++++-------- spec/status.md | 9 +- tools/ci/src/main.rs | 163 +++++++++--- tools/ci/tests/integration/cli_parts.rs | 31 +++ tools/ci/tests/integration/features.rs | 9 +- tools/ci/tests/integration/main.rs | 1 + tools/ci/tests/integration/target_dirs.rs | 12 +- tools/ci/tests/integration/test_first.rs | 89 ++++++- tools/ci/tests/integration/workflow.rs | 2 +- 14 files changed, 429 insertions(+), 183 deletions(-) create mode 100644 tools/ci/tests/integration/cli_parts.rs diff --git a/.github/rulesets/contextful-gate.proposed.json b/.github/rulesets/contextful-gate.proposed.json index 311691469..26b2796bb 100644 --- a/.github/rulesets/contextful-gate.proposed.json +++ b/.github/rulesets/contextful-gate.proposed.json @@ -18,9 +18,10 @@ { "context": "flare-dispatch/check:pins" }, { "context": "flare-dispatch/check:toolchain" }, { "context": "flare-dispatch/check:schema" }, - { "context": "flare-dispatch/check:test-first" }, + { "context": "flare-dispatch/check:test-first.validate" }, { "context": "flare-dispatch/check:workspace.compile" }, { "context": "flare-dispatch/check:workspace.cli" }, + { "context": "flare-dispatch/check:workspace.cli-formal" }, { "context": "flare-dispatch/check:workspace.foundation" }, { "context": "flare-dispatch/check:workspace.runtime" }, { "context": "flare-dispatch/check:workspace.read" }, @@ -29,9 +30,13 @@ { "context": "flare-dispatch/check:evaluate" }, { "context": "flare-dispatch/check:features.packages" }, { "context": "flare-dispatch/check:features.binary-none" }, + { "context": "flare-dispatch/check:features.formal-none" }, { "context": "flare-dispatch/check:features.binary-all" }, + { "context": "flare-dispatch/check:features.formal-all" }, { "context": "flare-dispatch/check:features.binary-data-plane" }, + { "context": "flare-dispatch/check:features.formal-data-plane" }, { "context": "flare-dispatch/check:features.binary-contextful-edge" }, + { "context": "flare-dispatch/check:features.formal-contextful-edge" }, { "context": "flare-dispatch/check:crate-graph" }, { "context": "flare-dispatch/check:connectors" }, { "context": "flare-dispatch/check:surfaces" }, diff --git a/AGENTS.md b/AGENTS.md index 0ba646572..ec682d94a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -121,21 +121,25 @@ The gate measures commits, so commit before running it. The FlareDispatch GitHub App dispatches `contextful-ci gate` from same-repository pull-request heads. Its `contextful-gate` run publishes the -`flare-dispatch/contextful-gate` parent and 24 child check-runs: -`flare-dispatch/check:` for each of `pins`, `toolchain`, `schema`, `test-first`, +`flare-dispatch/contextful-gate` parent, 29 static child check-runs, and one +test-first child per changed package: +`flare-dispatch/check:` for each of `pins`, `toolchain`, `schema`, `workspace`, `acceptance`, `evaluate`, `features`, `crate-graph`, `connectors`, -`surfaces`, `formal` and `budget`. The features and budget stages dispatch one check per -part, `flare-dispatch/check:features.` and `flare-dispatch/check:budget.`, so -each fits the sandbox's wall clock; `contextful-ci stages --parts` prints the list, and +`surfaces`, `formal` and `budget`. The test-first, features and budget stages dispatch one check per +part, `flare-dispatch/check:test-first.validate` and `flare-dispatch/check:test-first.`, `flare-dispatch/check:features.` and `flare-dispatch/check:budget.`, so +each fits the sandbox's wall clock; `contextful-ci stages --parts --base ` prints the list, and `--stage .` runs one part. A selected subset runs in that order and refuses a stage whose predecessor's output is absent; `--predecessors` runs those too. A local run and -the remote check invoke the identical command. `contextful-ci`'s suite asserts 24 -dispatchable parts. FlareDispatch runs `contextful-measures` nightly against the default +the remote check invoke the identical command. `contextful-ci`'s suite asserts 29 +static dispatchable parts. FlareDispatch runs `contextful-measures` nightly against the default branch and attaches its report to `refs/notes/measures`. A `v*` tag starts `contextful-release`, with `contextful-release-cell` for the ten cells in `contextful-ci release --plan`, `contextful-release-formula` for formulae and SHA256SUMS, and three independently tagged container images. The disabled -ruleset proposal under `.github/rulesets/` lists the parent and all 24 children. +ruleset proposal under `.github/rulesets/` lists the parent and all 29 static children. + +The CLI differential module runs in `workspace.cli-formal` and `features.formal-`; +`workspace.cli` and `features.binary-` run the remaining CLI tests. ### Landing without a remote verdict diff --git a/spec/81-engineering.md b/spec/81-engineering.md index d18451161..1bfad4edd 100644 --- a/spec/81-engineering.md +++ b/spec/81-engineering.md @@ -148,7 +148,7 @@ Target directories, the engine-linked invocation, linked query functions, build *because every engine-linked test binary embeds the bundled engine, and full debug information multiplies each past a sandbox's disk* - `release-profile` — Release builds compile with thin link-time optimization, one codegen unit per crate and symbols stripped, and unwind on panic. *because the run keeper survives a panicking job only by unwinding to its guard* -- `profile-build` — The features stage tests each package under every feature set its manifest lists in `feature-runs`, in parts dispatched as one check each: the other packages, and each binary run — none, all and each listed set. +- `profile-build` — The features stage tests each package under every feature set its manifest lists in `feature-runs`: other packages share one check; each binary feature run has separate checks for differential tests and all remaining tests. *because one cold build of the binary under one feature set fills most of a stage's wall clock, and a listed set reaches what none and all miss* - `container-image` — The repository's `Dockerfile` builds one profile, `contextful-full` unless `PROFILE` names another, as a static `linux/amd64` binary, and ships it in a shell-free runtime image as a non-root user over a declared store volume. - `targets` — All three profiles cross-compile to `x86_64-unknown-linux-musl` and `aarch64-unknown-linux-musl`; edge and full also build for `aarch64-apple-darwin` and `x86_64-apple-darwin`. @@ -184,15 +184,17 @@ Stage order, secrets of record, the crate-graph, row-token, egress and dependenc - `stage-sequence` — The gate runs its stages in order — pins, toolchain, schema, test-first, workspace, acceptance, evaluate, features, crate graph, connectors, TypeScript surfaces, formal, budget — and a subset is selectable by name. - `stage-subset` — A selected subset runs in the sequence's order; a stage reading an unselected predecessor's output, with that output absent, raises `StagePredecessorMissing`, naming both stages, before any stage runs. *A-assurance* -- `remote-check` — The FlareDispatch pull-request webhook dispatches every part from `contextful-ci stages --parts`, each as `flare-dispatch/check:` on the head commit. +- `remote-check` — The FlareDispatch pull-request webhook dispatches every part from `contextful-ci stages --parts --base `, each as `flare-dispatch/check:` on the head commit. *A-assurance* - `windows-checks` — Windows native checks run only as optional diagnostics and contribute no required merge or release verdict. *A-assurance* -- `workspace-parts` — Remote workspace checks compile the feature-unified workspace without running tests, run the CLI suite in its own check, and run each other non-acceptance package suite in exactly one of four groups. +- `test-first-parts` — Remote validation requires a changed test in each changed source package under {{assurance.test.test-first}}; each changed source or test package checks only its own tests against the base in a separate dispatch. + *because one base build per package fits the sandbox's wall clock more reliably* +- `workspace-parts` — Remote workspace checks compile the feature-unified workspace without running tests, run differential and remaining CLI tests separately, and run each other non-acceptance package suite in exactly one of four groups. *A-assurance* - `remote-predecessors` — Each dispatched part invokes `contextful-ci gate --predecessors --stage --base `, so no check reads another check's sandbox. *A-assurance* -- `fork-dispatch` — FlareDispatch dispatches only a pull-request head pushed to this repository; a fork receives none of the 24 required stage checks. +- `fork-dispatch` — FlareDispatch dispatches only a pull-request head pushed to this repository; a fork receives none of the required stage checks. *because a dispatch runs untrusted code in the organization's compute account, and absent required checks fail closed* - `stage-reports` — Each stage prints the environment it leaves and its memory limit, peak and event counts, and a failing stage prints its diagnostics before propagating its exit code. *because memory exhaustion is silent, and a kill then reads as a number in the log* diff --git a/spec/adr/A-assurance.md b/spec/adr/A-assurance.md index 25e2f6081..2d0f42604 100644 --- a/spec/adr/A-assurance.md +++ b/spec/adr/A-assurance.md @@ -144,16 +144,16 @@ Revisit: an external consumer needs a compatibility promise on a public surface, Context: the feature-unified workspace suite exceeds the 1800 s sandbox cap on a cold remote build. Criteria: every package suite runs, the feature union compiles, and each dispatched command runs locally too. -Decision: `assurance.gate.workspace-parts` dispatches one compile part over the full union and runs the CLI suite from its compiled artifacts. Four other parts partition the remaining non-acceptance package suites. The unsplit local workspace stage retains its one cargo invocation under `assurance.build.one-engine-build`. +Decision: `assurance.gate.workspace-parts` compiles the full union without execution and runs differential and remaining CLI tests in separate checks. Four other parts partition the remaining non-acceptance package suites. Binary feature checks use the same test partition. The unsplit local workspace stage retains its one cargo invocation under `assurance.build.one-engine-build`. | Option | Lost on | Cost | | --- | --- | --- | -| One union compile with CLI suite and four other suite groups *(chosen)* | — | Remote sandboxes rebuild shared dependencies; five checks replace one. | +| One union compile, two CLI partitions and four other suite groups *(chosen)* | — | Remote sandboxes rebuild shared dependencies; seven workspace checks replace one. | | A lock-keyed shared build cache | Operability | A cache bucket and credential become part of the gate; a cold lock still exceeds the cap. | | One check for the full workspace | Stage wall clock | A cold sandbox times out before it reports a verdict. | | A higher step timeout | Platform limit | Workflow steps admit at most 30 minutes. | -Consequences: the five remote checks cover the same package set as the unsplit local stage. The compile part reuses CLI artifacts before its sandbox exits; other parts compile their dependencies independently. +Consequences: seven remote workspace checks cover the same package set as the unsplit local stage. Every binary feature set retains all tests across its two checks. Each CLI partition compiles independently; reference-program builds occupy only the differential partition. ## One decision module compiles native and to WebAssembly diff --git a/spec/cards/assurance.md b/spec/cards/assurance.md index c8e4ed4b9..cddc56b94 100644 --- a/spec/cards/assurance.md +++ b/spec/cards/assurance.md @@ -16,7 +16,7 @@ Generated by `contextful-spec state` from the contract's files and `spec/terms/a | `automate` | Typed subcommands over compiled binaries, and the boundary shell keeps. | 5 | | `test` | Assertion construction, guard validation, suite placement, test-first and acceptance tests, and the connector test kit. | 16 | | `build` | Target directories, the engine-linked invocation, linked query functions, build targets, release artifacts and the dependency allowlist. | 20 | -| `gate` | Stage order, secrets of record, the crate-graph, row-token, egress and dependency rules, the formal stage, the container's ceilings, disk, footprint budgets and surface checks. | 39 | +| `gate` | Stage order, secrets of record, the crate-graph, row-token, egress and dependency rules, the formal stage, the container's ceilings, disk, footprint budgets and surface checks. | 40 | | `evaluate` | The read-path quality harness: case format, deterministic metrics and floors, judged dimensions, the reader and the judge. | 29 | | `baseline` | Holding a run against committed baselines and absolute floors, intervals, golden custody and case generation. | 23 | | `measure` | The target ledger: each tracked target, the clause it serves, how it is measured and its record per commit. | 18 | diff --git a/spec/spec.lock.json b/spec/spec.lock.json index 8f1c8e662..d7c5e59c5 100644 --- a/spec/spec.lock.json +++ b/spec/spec.lock.json @@ -22107,7 +22107,7 @@ "operation": "build", "subject": "profile-build", "kind": "behavior", - "statement": "The features stage tests each package under every feature set its manifest lists in `feature-runs`, in parts dispatched as one check each: the other packages, and each binary run — none, all and each listed set.", + "statement": "The features stage tests each package under every feature set its manifest lists in `feature-runs`: other packages share one check; each binary feature run has separate checks for differential tests and all remaining tests.", "why": "because one cold build of the binary under one feature set fills most of a stage's wall clock, and a listed set reaches what none and all miss", "file": "spec/81-engineering.md", "line": 151 @@ -22259,7 +22259,7 @@ "operation": "gate", "subject": "remote-check", "kind": "behavior", - "statement": "The FlareDispatch pull-request webhook dispatches every part from `contextful-ci stages --parts`, each as `flare-dispatch/check:` on the head commit.", + "statement": "The FlareDispatch pull-request webhook dispatches every part from `contextful-ci stages --parts --base `, each as `flare-dispatch/check:` on the head commit.", "why": "A-assurance", "file": "spec/81-engineering.md", "line": 187 @@ -22275,16 +22275,27 @@ "file": "spec/81-engineering.md", "line": 189 }, + { + "id": "assurance.gate.test-first-parts", + "contract": "assurance", + "operation": "gate", + "subject": "test-first-parts", + "kind": "behavior", + "statement": "Remote validation requires a changed test in each changed source package under {{assurance.test.test-first}}; each changed source or test package checks only its own tests against the base in a separate dispatch.", + "why": "because one base build per package fits the sandbox's wall clock more reliably", + "file": "spec/81-engineering.md", + "line": 191 + }, { "id": "assurance.gate.workspace-parts", "contract": "assurance", "operation": "gate", "subject": "workspace-parts", "kind": "behavior", - "statement": "Remote workspace checks compile the feature-unified workspace without running tests, run the CLI suite in its own check, and run each other non-acceptance package suite in exactly one of four groups.", + "statement": "Remote workspace checks compile the feature-unified workspace without running tests, run differential and remaining CLI tests separately, and run each other non-acceptance package suite in exactly one of four groups.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 191 + "line": 193 }, { "id": "assurance.gate.remote-predecessors", @@ -22295,7 +22306,7 @@ "statement": "Each dispatched part invokes `contextful-ci gate --predecessors --stage --base `, so no check reads another check's sandbox.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 193 + "line": 195 }, { "id": "assurance.gate.fork-dispatch", @@ -22303,10 +22314,10 @@ "operation": "gate", "subject": "fork-dispatch", "kind": "behavior", - "statement": "FlareDispatch dispatches only a pull-request head pushed to this repository; a fork receives none of the 24 required stage checks.", + "statement": "FlareDispatch dispatches only a pull-request head pushed to this repository; a fork receives none of the required stage checks.", "why": "because a dispatch runs untrusted code in the organization's compute account, and absent required checks fail closed", "file": "spec/81-engineering.md", - "line": 195 + "line": 197 }, { "id": "assurance.gate.stage-reports", @@ -22317,7 +22328,7 @@ "statement": "Each stage prints the environment it leaves and its memory limit, peak and event counts, and a failing stage prints its diagnostics before propagating its exit code.", "why": "because memory exhaustion is silent, and a kill then reads as a number in the log", "file": "spec/81-engineering.md", - "line": 197 + "line": 199 }, { "id": "assurance.gate.stage-timings", @@ -22328,7 +22339,7 @@ "statement": "Each dispatched part records queue, setup, compilation and test durations separately, alongside its commit, feature set, runner identity and cold or warm cache state.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 199, + "line": 201, "scenarios": [ "WHEN a part queues and then executes, THEN its report distinguishes queue, setup, compilation and test durations." ] @@ -22342,7 +22353,7 @@ "statement": "Every dispatched part completes within {{assurance.gate.container}} on cold and warm builds; oversized work splits into locally runnable parts retaining every required suite and predecessor obligation.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 201, + "line": 203, "scenarios": [ "WHEN every dispatched part runs with empty caches and then warm caches, THEN each completes within {{assurance.gate.container}} and the union preserves required coverage." ] @@ -22356,7 +22367,7 @@ "statement": "The runner admits concurrent builds only within available build slots and {{assurance.gate.parallel-jobs}}; excess builds queue before setup and compilation.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 203, + "line": 205, "scenarios": [ "WHEN occupied build slots reach the admission limit, THEN another build queues without starting setup or compilation." ] @@ -22370,7 +22381,7 @@ "statement": "A timed-out part retains phase timings and stdout/stderr diagnostics, terminates its owned process tree, verifies descendant exit and reports a failed verdict before releasing its build slot.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 205, + "line": 207, "scenarios": [ "WHEN a forced hang leaves a descendant holding output pipes, THEN timeout diagnostics survive, the descendant exits and the part fails before its slot is released." ] @@ -22384,7 +22395,7 @@ "statement": "The pins stage resolves every pinned artifact identity a run depends on before any compilation.", "why": "", "file": "spec/81-engineering.md", - "line": 207 + "line": 209 }, { "id": "assurance.gate.evaluate-stage", @@ -22395,7 +22406,7 @@ "statement": "The evaluate stage runs every gate-tier ledger entry and the native case set in the deterministic tier, and reports the floor and baseline verdicts.", "why": "", "file": "spec/81-engineering.md", - "line": 208 + "line": 210 }, { "id": "assurance.gate.schema-stage", @@ -22406,7 +22417,7 @@ "statement": "The schema stage regenerates each derived artifact into a scratch location, compares it byte for byte against the committed copy, and runs `contextful-spec lint`.", "why": "", "file": "spec/81-engineering.md", - "line": 209 + "line": 211 }, { "id": "assurance.gate.secret-ciphertext", @@ -22417,7 +22428,7 @@ "statement": "In the schema stage, a key other than `DOTENV_PUBLIC_KEY*` in a git-tracked `.env*` file other than `.env.example` holding a value without the `encrypted:` prefix, or a tracked `.env.keys`, raises `SecretPlaintext`, naming file and key.", "why": "because a tracked file reaches every clone, and only dotenvx ciphertext is safe there while its private key stays untracked", "file": "spec/81-engineering.md", - "line": 210 + "line": 212 }, { "id": "assurance.gate.secret-scope", @@ -22428,7 +22439,7 @@ "statement": "In the schema stage, a key other than `DOTENV_PUBLIC_KEY*` in a git-tracked `.env*` file whose block — the assignments above it, up to the first blank line — opens with no comment carrying text raises `SecretScopeMissing`, naming file and key.", "why": "because a provider token cannot report its own grants, so the comment above it is the one record of what it reaches", "file": "spec/81-engineering.md", - "line": 212 + "line": 214 }, { "id": "assurance.gate.crate-graph", @@ -22439,7 +22450,7 @@ "statement": "The crate-graph stage holds run-path crates to reaching read-path crates through the three crossing crates alone; another edge raises `CrateGraphViolation`, naming both crates.", "why": "P5", "file": "spec/81-engineering.md", - "line": 214 + "line": 216 }, { "id": "assurance.gate.locked-resolve", @@ -22450,7 +22461,7 @@ "statement": "The crate-graph stage resolves every graph `--locked` against the committed `Cargo.lock`, so a lock behind its manifests, or one rewritten before the stage starts, fails the stage before any rule runs; the stage never rewrites it.", "why": "because a rule held over a graph the lock file does not record passes a build that resolves another", "file": "spec/81-engineering.md", - "line": 216 + "line": 218 }, { "id": "assurance.gate.row-token", @@ -22461,7 +22472,7 @@ "statement": "Every row-returning function of the store crate takes the enforcement token type, whose constructor is private to the enforcement module; a row path outside a registered relation raises `EnforceUnmediatedPath`, naming the function.", "why": "P5", "file": "spec/81-engineering.md", - "line": 218 + "line": 220 }, { "id": "assurance.gate.dependency-deny", @@ -22472,7 +22483,7 @@ "statement": "The crate-graph stage runs pinned cargo-deny `check bans` over each profile's resolved graph, denying every crate a profile-wide dependency refusal names; a hit raises that refusal's error, naming the profile and the path.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 220 + "line": 222 }, { "id": "assurance.gate.deny-profile-graph", @@ -22483,7 +22494,7 @@ "statement": "A profile's graph is the binary resolved with default features off and the profile's feature on, across every target, excluding development dependencies; the stage prints each profile the binary declares no feature for.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 222 + "line": 224 }, { "id": "assurance.gate.deny-list", @@ -22494,7 +22505,7 @@ "statement": "`deny.toml` holds one `[bans]` entry per crate or feature a profile-wide dependency refusal names, its reason that refusal's clause id; an entry missing or extra raises `DenyListDrift`, naming it.", "why": "because a refusal whose crate the list omits passes every profile graph linking it", "file": "spec/81-engineering.md", - "line": 224 + "line": 226 }, { "id": "assurance.gate.unconfigured-egress", @@ -22505,7 +22516,7 @@ "statement": "An outbound call reachable under the default configuration — usage ping, license check, update probe — raises `UnconfiguredEgress` in the crate-graph stage, naming the call site.", "why": "because some deployments run with no external reach, and a call nobody configured moves data outside every grant, zone and audit record", "file": "spec/81-engineering.md", - "line": 226 + "line": 228 }, { "id": "assurance.gate.interpolated-claim", @@ -22516,7 +22527,7 @@ "statement": "A source lint over the runtime crates finding a subject claim formatted into SQL text raises `EnforceInterpolatedSubjectClaim`, naming the file and line.", "why": "P1", "file": "spec/81-engineering.md", - "line": 228 + "line": 230 }, { "id": "assurance.gate.formal-stage", @@ -22527,7 +22538,7 @@ "statement": "The formal stage runs {{assurance.audit-assumptions.check-command}}, {{assurance.differential-test.command}} and {{assurance.model.protocol-check}}; a non-zero exit from any reds the run.", "why": "P7", "file": "spec/81-engineering.md", - "line": 230 + "line": 232 }, { "id": "assurance.gate.container", @@ -22538,7 +22549,7 @@ "statement": "The gate container carries four ceilings — processor count, 12 GiB of memory, 18 GiB of usable disk, and a 30 min per-stage wall clock — and a run dies on any one.", "why": "", "file": "spec/81-engineering.md", - "line": 232 + "line": 234 }, { "id": "assurance.gate.parallel-jobs", @@ -22549,7 +22560,7 @@ "statement": "Parallel build jobs number the memory ceiling divided by 3 GiB.", "why": "", "file": "spec/81-engineering.md", - "line": 233 + "line": 235 }, { "id": "assurance.gate.free-disk", @@ -22560,7 +22571,7 @@ "statement": "A stage starting with less than 2 GiB of free disk raises `BuildDiskPrecondition` and exits 28 before doing work.", "why": "P7", "file": "spec/81-engineering.md", - "line": 234 + "line": 236 }, { "id": "assurance.gate.build-size", @@ -22571,7 +22582,7 @@ "statement": "The budget stage holds total build size to 12 GiB, printing that size, the free space and the largest artifacts; a larger build raises `BuildSizeCeilingExceeded`, naming the head of that list.", "why": "P7", "file": "spec/81-engineering.md", - "line": 236 + "line": 238 }, { "id": "assurance.gate.edge-budget", @@ -22582,7 +22593,7 @@ "statement": "The edge profile holds to 50 MiB compressed and 60 MiB idle resident set.", "why": "", "file": "spec/81-engineering.md", - "line": 238 + "line": 240 }, { "id": "assurance.gate.full-budget", @@ -22593,7 +22604,7 @@ "statement": "The full profile holds to 150 MiB compressed and 180 MiB idle resident set.", "why": "", "file": "spec/81-engineering.md", - "line": 239 + "line": 241 }, { "id": "assurance.gate.control-budget", @@ -22604,7 +22615,7 @@ "statement": "The control profile holds to 60 MiB compressed and 80 MiB idle resident set.", "why": "", "file": "spec/81-engineering.md", - "line": 240 + "line": 242 }, { "id": "assurance.gate.footprint", @@ -22615,7 +22626,7 @@ "statement": "Per change and per profile, the footprint step builds the static-linked Linux target, compresses it, and holds its size to the profile's budget and its dynamic dependencies to the platform C library.", "why": "P7", "file": "spec/81-engineering.md", - "line": 241 + "line": 243 }, { "id": "assurance.gate.budget-stage", @@ -22626,7 +22637,7 @@ "statement": "The budget stage runs {{assurance.gate.footprint}} over every profile, in one part per profile dispatched as its own check, and the evaluate stage builds no profile.", "why": "because one link-time-optimized release build of the full profile fills most of one stage's wall clock", "file": "spec/81-engineering.md", - "line": 243 + "line": 245 }, { "id": "assurance.gate.footprint-exceeded", @@ -22637,7 +22648,7 @@ "statement": "An artifact over its profile's budget, or carrying a dynamic dependency beyond the platform C library, raises `FootprintBudgetExceeded`, naming the profile.", "why": "P7", "file": "spec/81-engineering.md", - "line": 245 + "line": 247 }, { "id": "assurance.gate.typescript-surfaces", @@ -22648,7 +22659,7 @@ "statement": "The TypeScript surfaces run each declared typecheck, test and build script in one stage; an absent script skips only that script.", "why": "", "file": "spec/81-engineering.md", - "line": 247 + "line": 249 }, { "id": "assurance.gate.native-test-files", @@ -22659,7 +22670,7 @@ "statement": "The surfaces stage runs every regular native `.test.ts` file under a direct TypeScript surface's `test/` or `tests/` tree through Node's TypeScript strip runner, and refuses absent passing tests or pinned names.", "why": "because a package script can omit a pinned test while reporting success", "file": "spec/81-engineering.md", - "line": 248 + "line": 250 }, { "id": "assurance.gate.surface-check-failed", @@ -22670,7 +22681,7 @@ "statement": "A surface whose typecheck, unit tests or framework build fails raises `SurfaceCheckFailed`, naming the surface and the script.", "why": "P7", "file": "spec/81-engineering.md", - "line": 250 + "line": 252 }, { "id": "assurance.evaluate.real-read-path", @@ -22681,7 +22692,7 @@ "statement": "An evaluation run ingests, indexes, retrieves, reads, judges and scores through the surfaces a caller uses, and the harness adds no storage or runtime primitive.", "why": "", "file": "spec/81-engineering.md", - "line": 304 + "line": 306 }, { "id": "assurance.evaluate.through-the-store", @@ -22692,7 +22703,7 @@ "statement": "The corpus loads through the real store, and the retriever under test calls {{read.retrieve.ranked-call}} with the options a caller passes.", "why": "because an unordered full-scan retriever cannot go red when ranking breaks", "file": "spec/81-engineering.md", - "line": 305 + "line": 307 }, { "id": "assurance.evaluate.policy-labels", @@ -22703,7 +22714,7 @@ "statement": "An evaluation corpus carries zone and row-policy labels in its table declarations, and a run reads it through the access-control path under one admitted credential.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 307, + "line": 309, "scenarios": [ "WHEN a regression case names a row another agent owns under `owner = subject.agent`, THEN no leg returns it and the hybrid forbidden-row rate reads 0." ] @@ -22717,7 +22728,7 @@ "statement": "A run over a corpus whose tables declare no zone or row policy, with no `local:` zone passed as `--zone`, raises `EvalCorpusUnlabeled` before any row lands.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 309, + "line": 311, "scenarios": [ "WHEN a corpus's tables declare no policy and the run passes no `--zone`, THEN the run raises `EvalCorpusUnlabeled` and lands no row." ] @@ -22731,7 +22742,7 @@ "statement": "A case carries an id, a corpus path relative to its case file, a question, tags, and an expected block of answer, artifacts, edges, entities, must-cite, must-abstain, must-not-retrieve, time anchor and recency bound.", "why": "", "file": "spec/81-engineering.md", - "line": 311 + "line": 313 }, { "id": "assurance.evaluate.row-reference", @@ -22742,7 +22753,7 @@ "statement": "A case names a row as `#`, the key joining the row's declared primary-key values with commas, and the runner keys each returned row the same way.", "why": "", "file": "spec/81-engineering.md", - "line": 312 + "line": 314 }, { "id": "assurance.evaluate.regression-case", @@ -22753,7 +22764,7 @@ "statement": "A case tagged `regression` is a regression case, and its must-not-retrieve set scores {{assurance.evaluate.forbidden-row-rate}}.", "why": "", "file": "spec/81-engineering.md", - "line": 313 + "line": 315 }, { "id": "assurance.evaluate.corpus-layout", @@ -22764,7 +22775,7 @@ "statement": "A corpus is a directory holding `contextful.toml`, its table declarations and policy labels, and one `rows/
.jsonl` per table.", "why": "", "file": "spec/81-engineering.md", - "line": 314 + "line": 316 }, { "id": "assurance.evaluate.run-command", @@ -22775,7 +22786,7 @@ "statement": "`contextful eval run --goldens ` loads the case file, lands and folds each referenced corpus into a scratch store at `--landed-at`, the Unix epoch by default, reads every case, and writes the run report.", "why": "because an undated row landed at the epoch predates every question's window, so a replay scores alike on any day", "file": "spec/81-engineering.md", - "line": 315 + "line": 317 }, { "id": "assurance.evaluate.legs", @@ -22786,7 +22797,7 @@ "statement": "Each case calls {{read.retrieve.ranked-call}} three times at limit k: the lexical leg with the question alone, the vector leg with an embedding and an empty query, the hybrid leg with both.", "why": "", "file": "spec/81-engineering.md", - "line": 317 + "line": 319 }, { "id": "assurance.evaluate.stub-embedder", @@ -22797,7 +22808,7 @@ "statement": "The deterministic tier embeds every row and question with a seeded feature-hashing stub embedder; a corpus row or a case carrying its own embedding keeps it.", "why": "", "file": "spec/81-engineering.md", - "line": 318 + "line": 320 }, { "id": "assurance.evaluate.run-verdict", @@ -22808,7 +22819,7 @@ "statement": "A run holds its report to the floors and, given `--baseline`, to that file; either red verdict exits non-zero, and `--update-baseline` applies {{assurance.baseline.raise-only}} on green alone.", "why": "", "file": "spec/81-engineering.md", - "line": 319 + "line": 321 }, { "id": "assurance.evaluate.converter", @@ -22819,7 +22830,7 @@ "statement": "Each benchmark and application ground truth converts once into the case format through its own converter, and the runner reads nothing else.", "why": "", "file": "spec/81-engineering.md", - "line": 320 + "line": 322 }, { "id": "assurance.evaluate.deterministic-tier", @@ -22830,7 +22841,7 @@ "statement": "Recall@k, R-precision, hit-rate@k, reciprocal rank and nDCG@k run with no model call, reported per leg — lexical, vector, hybrid — and per slice.", "why": "", "file": "spec/81-engineering.md", - "line": 321 + "line": 323 }, { "id": "assurance.evaluate.distinct-top-k", @@ -22841,7 +22852,7 @@ "statement": "Membership in the top k is distinct, and a repeated id counts once.", "why": "", "file": "spec/81-engineering.md", - "line": 322 + "line": 324 }, { "id": "assurance.evaluate.absent-truth", @@ -22852,7 +22863,7 @@ "statement": "A ranked metric over a case with no relevant set is NaN and drops out of every aggregate; an empty ranking against a non-empty relevant set scores zero.", "why": "", "file": "spec/81-engineering.md", - "line": 323 + "line": 325 }, { "id": "assurance.evaluate.precision-floor", @@ -22863,7 +22874,7 @@ "statement": "Precision at rank min(k, R), R being the relevant-set size, holds at or above 60 percent.", "why": "because precision@k caps at R/k, below the floor whenever R is small", "file": "spec/81-engineering.md", - "line": 324 + "line": 326 }, { "id": "assurance.evaluate.forbidden-row-rate", @@ -22874,7 +22885,7 @@ "statement": "A row named in a must-not-retrieve set holds at 0 percent of a regression case's returned rows.", "why": "", "file": "spec/81-engineering.md", - "line": 326 + "line": 328 }, { "id": "assurance.evaluate.duplicate-row-rate", @@ -22885,7 +22896,7 @@ "statement": "A repeated table-and-row-key pair holds at 0 percent of the ranking.", "why": "", "file": "spec/81-engineering.md", - "line": 327 + "line": 329 }, { "id": "assurance.evaluate.in-window-rate", @@ -22896,7 +22907,7 @@ "statement": "A case declaring a recency bound holds its in-window rate at or above 95 percent.", "why": "", "file": "spec/81-engineering.md", - "line": 328 + "line": 330 }, { "id": "assurance.evaluate.abstention", @@ -22907,7 +22918,7 @@ "statement": "A must-abstain case returns zero rows.", "why": "", "file": "spec/81-engineering.md", - "line": 329 + "line": 331 }, { "id": "assurance.evaluate.truth-per-surface", @@ -22918,7 +22929,7 @@ "statement": "Expected artifacts score the artifact retriever and expected edges score the edge retriever, reported apart; edge truth with no edge surface configured counts in an unscored tally.", "why": "", "file": "spec/81-engineering.md", - "line": 330 + "line": 332 }, { "id": "assurance.evaluate.judged-dimensions", @@ -22929,7 +22940,7 @@ "statement": "The reading stage adds four judged dimensions: answer accuracy; abstention, as correct-refusal and hallucination-on-unknown rates; citation faithfulness; and temporal correctness against the bitemporal columns.", "why": "", "file": "spec/81-engineering.md", - "line": 331 + "line": 333 }, { "id": "assurance.evaluate.grounded-reader", @@ -22940,7 +22951,7 @@ "statement": "The harness reader answers from retrieved rows alone, and the deterministic tier substitutes a stub reader.", "why": "", "file": "spec/81-engineering.md", - "line": 332 + "line": 334 }, { "id": "assurance.evaluate.model-endpoint", @@ -22951,7 +22962,7 @@ "statement": "The judge and the reader reach a model through {{connector.infer.model-endpoint}}, and the harness holds no credential.", "why": "", "file": "spec/81-engineering.md", - "line": 333 + "line": 335 }, { "id": "assurance.evaluate.judge", @@ -22962,7 +22973,7 @@ "statement": "The judge is one pinned open-weights instruct model, called once per judged item at temperature zero; a judge swap is a full re-baseline.", "why": "because a majority vote at temperature zero samples one mode and adds cost without reducing variance", "file": "spec/81-engineering.md", - "line": 334 + "line": 336 }, { "id": "assurance.evaluate.systems-metrics", @@ -22973,7 +22984,7 @@ "statement": "Tokens per query, latency and cost report beside the quality figures, bucketed by corpus size in tokens relative to the reader's context window.", "why": "", "file": "spec/81-engineering.md", - "line": 336 + "line": 338 }, { "id": "assurance.evaluate.checkpoint", @@ -22984,7 +22995,7 @@ "statement": "A run appends each case's result as JSONL, and a crash re-runs the in-flight case.", "why": "", "file": "spec/81-engineering.md", - "line": 337 + "line": 339 }, { "id": "assurance.evaluate.run-report", @@ -22995,7 +23006,7 @@ "statement": "A run report carries one field per gated metric, the sample count behind each mean, the run block, the per-slice breakdown, and the tally of cases sampled, dropped and unscored.", "why": "", "file": "spec/81-engineering.md", - "line": 338 + "line": 340 }, { "id": "assurance.baseline.file", @@ -23006,7 +23017,7 @@ "statement": "A baseline file carries a reserved run block `{k, tier, model, samples}` and one entry per gated metric: a bare number at the run's dead band, or an object overriding the band.", "why": "", "file": "spec/81-engineering.md", - "line": 349 + "line": 351 }, { "id": "assurance.baseline.default-dead-band", @@ -23017,7 +23028,7 @@ "statement": "A rate-valued entry with no override gates at a 2 percent dead band.", "why": "", "file": "spec/81-engineering.md", - "line": 350 + "line": 352 }, { "id": "assurance.baseline.rank-quality-dead-band", @@ -23028,7 +23039,7 @@ "statement": "The ranked-quality entry gates at a 3 percent dead band.", "why": "", "file": "spec/81-engineering.md", - "line": 351 + "line": 353 }, { "id": "assurance.baseline.band-units", @@ -23039,7 +23050,7 @@ "statement": "A band carries its metric's own units, and a count entry is pinned at zero with no band.", "why": "", "file": "spec/81-engineering.md", - "line": 352 + "line": 354 }, { "id": "assurance.baseline.metric-path", @@ -23050,7 +23061,7 @@ "statement": "An entry names a metric by the report's field path: `retrieval..`, `edge_retrieval..`, `judge.`, `latency_ms`, `n_cases`, or `slices..` before a bare `` or any of these but `latency_ms`.", "why": "", "file": "spec/81-engineering.md", - "line": 353 + "line": 355 }, { "id": "assurance.baseline.sample-count", @@ -23061,7 +23072,7 @@ "statement": "Every mean-valued path takes a `.n` suffix naming its sample count.", "why": "because blanking one case's truth lifts a NaN-filtered mean without moving the case count", "file": "spec/81-engineering.md", - "line": 354 + "line": 356 }, { "id": "assurance.baseline.unresolvable-path", @@ -23072,7 +23083,7 @@ "statement": "A path resolving to nothing, a mean over fewer than 30 cases, a malformed entry or a non-finite band raises `BaselinePathUnresolved` before the suite runs.", "why": "P7", "file": "spec/81-engineering.md", - "line": 356 + "line": 358 }, { "id": "assurance.baseline.run-stamp-drift", @@ -23083,7 +23094,7 @@ "statement": "A run block differing from the run's own configuration raises `BaselineRunStampMismatch` before the suite runs.", "why": "P7", "file": "spec/81-engineering.md", - "line": 358 + "line": 360 }, { "id": "assurance.baseline.interval", @@ -23094,7 +23105,7 @@ "statement": "Each judged figure reports a 95 percent percentile-bootstrap interval over cases from 1000 resamples.", "why": "", "file": "spec/81-engineering.md", - "line": 360 + "line": 362 }, { "id": "assurance.baseline.raise-only", @@ -23105,7 +23116,7 @@ "statement": "A baseline update runs after every gate and floor passes, moves each improved entry to its measured value, moves none in the worse direction, and adds no path.", "why": "", "file": "spec/81-engineering.md", - "line": 361 + "line": 363 }, { "id": "assurance.baseline.floors-are-absolute", @@ -23116,7 +23127,7 @@ "statement": "An absolute floor gates independently of every committed value.", "why": "", "file": "spec/81-engineering.md", - "line": 362 + "line": 364 }, { "id": "assurance.baseline.floor-coverage", @@ -23127,7 +23138,7 @@ "statement": "A report lacking a floor's figure on any leg of its `retrieval` surface breaches that floor; only a forbidden-row or in-window rate over no case holds.", "why": "", "file": "spec/81-engineering.md", - "line": 363 + "line": 365 }, { "id": "assurance.baseline.offline", @@ -23138,7 +23149,7 @@ "statement": "The gate is a local command comparing the report against in-tree baselines and floors, reporting both verdicts, with or without a trace store reachable.", "why": "", "file": "spec/81-engineering.md", - "line": 364 + "line": 366 }, { "id": "assurance.baseline.trace-store", @@ -23149,7 +23160,7 @@ "statement": "A trace store records run history and curation staging, and decides no verdict.", "why": "", "file": "spec/81-engineering.md", - "line": 365 + "line": 367 }, { "id": "assurance.baseline.golden-custody", @@ -23160,7 +23171,7 @@ "statement": "Version-controlled JSONL under `evals/cases/` is the canonical golden set, one loader is its only ingestion door, and a reviewer's edit reaches the gate as a change to the tree.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 366 + "line": 368 }, { "id": "assurance.baseline.answer-key-in-the-store", @@ -23171,7 +23182,7 @@ "statement": "An artifact path normalizing under the evaluation directory, reached by a manifest lint or a connector, raises `GoldenSetIngested`.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 368 + "line": 370 }, { "id": "assurance.baseline.generated-redaction", @@ -23182,7 +23193,7 @@ "statement": "A golden mined from a live query or promoted from a labeled outcome passes {{authority.redact.write-time}} before commit; a case it cannot redact raises `GoldenRedactionFailed`.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 370 + "line": 372 }, { "id": "assurance.baseline.promotion-source", @@ -23193,7 +23204,7 @@ "statement": "Only an adjudicated outcome label promotes into a golden set; a self-rated row does not.", "why": "", "file": "spec/81-engineering.md", - "line": 372 + "line": 374 }, { "id": "assurance.baseline.generators", @@ -23204,7 +23215,7 @@ "statement": "Three generators draft candidate cases from the deployment's store — entity-to-fact-to-edge walks, composed edge chains, and questions about absent entities — and a human approves each before commit.", "why": "", "file": "spec/81-engineering.md", - "line": 373 + "line": 375 }, { "id": "assurance.baseline.held-out", @@ -23215,7 +23226,7 @@ "statement": "Retrieval and synthesis are tuned on no gate input.", "why": "", "file": "spec/81-engineering.md", - "line": 374 + "line": 376 }, { "id": "assurance.baseline.rotation", @@ -23226,7 +23237,7 @@ "statement": "The native set rotates on a cadence, a sample of its truth is reviewed on each rotation, and a comparison across generations resolves through the pinned run stamp.", "why": "", "file": "spec/81-engineering.md", - "line": 375 + "line": 377 }, { "id": "assurance.baseline.native-gate", @@ -23237,7 +23248,7 @@ "statement": "A native benchmark generated from a real corpus is the red or green gate; public benchmark sets run beside it as held-out comparison and decide nothing.", "why": "", "file": "spec/81-engineering.md", - "line": 376 + "line": 378 }, { "id": "assurance.baseline.trace-export", @@ -23248,7 +23259,7 @@ "statement": "A run touching a deployed store with a hosted trace endpoint configured raises `TraceExportOutOfPerimeter`; hosted collection serves runs over public or synthetic fixtures alone.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 377 + "line": 379 }, { "id": "assurance.measure.ledger", @@ -23259,7 +23270,7 @@ "statement": "`evals/ledger.toml` holds one entry per tracked target: an id, an owning clause id, a run-report metric path, a tier, a method naming one integration test, case set or probe, and a threshold.", "why": "", "file": "spec/81-engineering.md", - "line": 386 + "line": 388 }, { "id": "assurance.measure.unresolved-entry", @@ -23270,7 +23281,7 @@ "statement": "An entry whose owning clause, method or metric path resolves to nothing raises `MeasureEntryUnresolved` before any measure runs.", "why": "P7", "file": "spec/81-engineering.md", - "line": 387, + "line": 389, "scenarios": [ "WHEN an entry names `run.journal.entry-keys`, THEN the run raises `MeasureEntryUnresolved` and no test runs." ] @@ -23284,7 +23295,7 @@ "statement": "An entry naming an issue in place of a method reports open and gates nothing, and `evals/ledger.md` carries every entry's computed status.", "why": "", "file": "spec/81-engineering.md", - "line": 389, + "line": 391, "scenarios": [ "WHEN an entry's method is `{ issue = 43 }`, THEN `evals/ledger.md` lists it open and the evaluate stage ignores it." ] @@ -23298,7 +23309,7 @@ "statement": "A gate-tier entry decides the evaluate stage, a trend-tier entry is measured on the default-branch run and decides nothing, and a scheduled-tier entry runs on the scheduled job alone.", "why": "", "file": "spec/81-engineering.md", - "line": 390 + "line": 392 }, { "id": "assurance.measure.count-first", @@ -23309,7 +23320,7 @@ "statement": "A gate-tier entry measures a count, a ratio within one run or a size under a locked resolve; a wall-clock or resident-memory figure is trend-tier.", "why": "because a shared container moves wall-clock figures past any band narrow enough to catch a regression", "file": "spec/81-engineering.md", - "line": 391 + "line": 393 }, { "id": "assurance.measure.record", @@ -23320,7 +23331,7 @@ "statement": "A measure writes one JSON record carrying its entry id, value, sample count, seed and run stamp; a gate-tier method finishing without one raises `MeasureRecordMissing`.", "why": "P7", "file": "spec/81-engineering.md", - "line": 393 + "line": 395 }, { "id": "assurance.measure.seeded", @@ -23331,7 +23342,7 @@ "statement": "Every generated fixture and randomized schedule derives from the record's seed, and replaying that seed reproduces a count-valued entry's value.", "why": "because a failure that cannot replay cannot be fixed", "file": "spec/81-engineering.md", - "line": 395 + "line": 397 }, { "id": "assurance.measure.seed-mismatch", @@ -23342,7 +23353,7 @@ "statement": "A gate or scheduled record whose seed differs from its entry's declared seed raises `MeasureSeedMismatch` and counts as red; a trend mismatch leaves no comparable record.", "why": "because a figure measured under another seed replays nothing the ledger names", "file": "spec/81-engineering.md", - "line": 397 + "line": 399 }, { "id": "assurance.measure.timing-iterations", @@ -23353,7 +23364,7 @@ "statement": "A timed batch reports p50 and p95 over 200 repeats of its operation, each after warm-up.", "why": "", "file": "spec/81-engineering.md", - "line": 399 + "line": 401 }, { "id": "assurance.measure.timing-batches", @@ -23364,7 +23375,7 @@ "statement": "A timed figure is the median of 5 repeats of its timed batch.", "why": "", "file": "spec/81-engineering.md", - "line": 400 + "line": 402 }, { "id": "assurance.measure.trend-direction", @@ -23375,7 +23386,7 @@ "statement": "Each trend-tier ledger entry declares `higher_is_better` or `lower_is_better` for its metric.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 401 + "line": 403 }, { "id": "assurance.measure.trend-baseline", @@ -23386,7 +23397,7 @@ "statement": "A trend figure compares with the newest earlier successful default-branch report for the same entry and seed under {{assurance.measure.runner-stamp}}; absent matching history yields no annotation.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 403, + "line": 405, "scenarios": [ "WHEN a newer failed report follows two successful matches, THEN the trend compares with the newer successful match and the stage passes." ] @@ -23400,7 +23411,7 @@ "statement": "A trend figure more than 25 percent worse than its baseline annotates the run report and fails no stage.", "why": "", "file": "spec/81-engineering.md", - "line": 405, + "line": 407, "scenarios": [ "WHEN a trend p95 moves from 40 ms to 52 ms on a matching runner stamp, THEN the report carries a +30 percent annotation and the stage passes." ] @@ -23414,7 +23425,7 @@ "statement": "The run block carries the runner's processor model, processor count and memory limit, and a trend figure compares only against a baseline with the same stamp.", "why": "", "file": "spec/81-engineering.md", - "line": 406 + "line": 408 }, { "id": "assurance.measure.absolute-threshold", @@ -23425,7 +23436,7 @@ "statement": "A threshold is an absolute figure of this system's own measure, and it tightens only through {{assurance.baseline.raise-only}}.", "why": "", "file": "spec/81-engineering.md", - "line": 407 + "line": 409 }, { "id": "assurance.measure.history", @@ -23436,7 +23447,7 @@ "statement": "A run on the default branch attaches its run report to the measured commit under `refs/notes/measures`, and no verdict reads a note.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 408 + "line": 410 }, { "id": "assurance.measure.scheduled", @@ -23447,7 +23458,7 @@ "statement": "The FlareDispatch nightly run executes every tier against the default-branch head and reports one check per run.", "why": "", "file": "spec/81-engineering.md", - "line": 410 + "line": 412 }, { "id": "assurance.measure.measured-basis", @@ -23458,7 +23469,7 @@ "statement": "A bound with a measured basis names a trend-tier or scheduled-tier ledger entry id as its benchmark.", "why": "", "file": "spec/81-engineering.md", - "line": 411 + "line": 413 }, { "id": "assurance.release.version", @@ -23469,7 +23480,7 @@ "statement": "A release tag is `v0..`: `` counts the milestones computing `closed` in the tagged commit's `spec/status.md`, and `` counts the existing tags `v0..*`.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 441, + "line": 443, "scenarios": [ "WHEN milestones 0 to 7 and 11 compute `closed` and `v0.9.0` exists, THEN the computed version is `v0.9.1`." ] @@ -23483,7 +23494,7 @@ "statement": "`contextful-ci tag` creates a signed annotated tag on `HEAD` naming the closed milestones once every refusal of this operation clears, and never moves or replaces an existing tag.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 443 + "line": 445 }, { "id": "assurance.release.dirty-tree", @@ -23494,7 +23505,7 @@ "statement": "A working tree or index differing from `HEAD` raises `TagTreeDirty`.", "why": "because the gate then judges files the tagged commit does not hold", "file": "spec/81-engineering.md", - "line": 445 + "line": 447 }, { "id": "assurance.release.off-branch", @@ -23505,7 +23516,7 @@ "statement": "A `HEAD` unreachable from the default branch, `origin/HEAD` unless `--branch` names another, raises `TagOffDefaultBranch`.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 447 + "line": 449 }, { "id": "assurance.release.version-regressed", @@ -23516,7 +23527,7 @@ "statement": "A computed version below the highest existing `v..` tag raises `TagVersionRegressed`, naming both.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 449, + "line": 451, "scenarios": [ "WHEN milestone 11 reopens after `v0.9.0`, THEN the computed `v0.8.0` raises `TagVersionRegressed`." ] @@ -23530,7 +23541,7 @@ "statement": "A `[workspace.package]` version in `HEAD`'s `Cargo.toml` that is absent or differs from the computed version raises `TagWorkspaceVersionMismatch`, naming both.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 451 + "line": 453 }, { "id": "assurance.release.gate-failed", @@ -23541,7 +23552,7 @@ "statement": "Once the other refusals clear, every gate stage runs against `HEAD` with `HEAD~1` as base; a failing stage raises `TagGateFailed`, naming the stage's refusal, and no tag is created.", "why": "A-assurance", "file": "spec/81-engineering.md", - "line": 453 + "line": 455 }, { "id": "assurance.release.release-run", @@ -23552,7 +23563,7 @@ "statement": "A `v*` tag at a default-branch commit starts FlareDispatch `contextful-release`, with one `contextful-release-cell` child per {{assurance.build.targets}} matrix cell and one `contextful-release-formula` child for their metadata.", "why": "because distributed builds and formula generation share one tag identity", "file": "spec/81-engineering.md", - "line": 455 + "line": 457 }, { "id": "assurance.release.release-dry-run", @@ -23563,7 +23574,7 @@ "statement": "A manual `contextful-release` dry run builds the release assets and publishes neither a GitHub release nor container images.", "why": "because a packaging rehearsal must not claim a released version", "file": "spec/81-engineering.md", - "line": 457 + "line": 459 } ], "owns": { @@ -25143,6 +25154,10 @@ "assurance.build.wasi-probe", "assurance.gate.edge-budget" ], + [ + "assurance.gate.test-first-parts", + "assurance.test.test-first" + ], [ "assurance.gate.part-budget", "assurance.gate.container" diff --git a/spec/status.md b/spec/status.md index ed7e15210..553454faf 100644 --- a/spec/status.md +++ b/spec/status.md @@ -5,7 +5,7 @@ An unpinned clause is `committed`; a pinned one is `performed` when its Rust, Le | Contract | Files | Operations | Clauses | Refusals | Limits | Unsettled | Performed | Broken | Floor | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| `assurance` | 2 | 15 | 245 | 59 | 22 | 14 | 141 | 0 | 138 | +| `assurance` | 2 | 15 | 246 | 59 | 22 | 14 | 142 | 0 | 138 | | `authority` | 2 | 16 | 269 | 77 | 27 | 10 | 206 | 0 | 206 | | `connector` | 2 | 14 | 349 | 89 | 21 | 15 | 229 | 0 | 228 | | `corpus` | 1 | 10 | 79 | 37 | 7 | 0 | 43 | 0 | 43 | @@ -15,7 +15,7 @@ An unpinned clause is `committed`; a pinned one is `performed` when its Rust, Le | `store` | 1 | 17 | 249 | 57 | 13 | 19 | 213 | 0 | 204 | | `surface` | 2 | 20 | 114 | 51 | 22 | 0 | 59 | 0 | 59 | | `topology` | 1 | 6 | 70 | 24 | 5 | 5 | 38 | 0 | 38 | -| **total** | 19 | 155 | 2115 | 593 | 191 | 103 | 1476 | 0 | | +| **total** | 19 | 155 | 2116 | 593 | 191 | 103 | 1477 | 0 | | Decision records: 18. @@ -39,7 +39,7 @@ A milestone reads `closed` when its acceptance test computes `passing` and every | 11 — The derive tier | 7 | 88 | 79 | passing | closed | | 12 — The operator console | 12 | 49 | 0 | open | open | | 13 — Disclosure | 5 | 33 | 13 | open | open | -| 14 — Assurance | 7 | 145 | 89 | passing | closed | +| 14 — Assurance | 7 | 146 | 90 | passing | closed | Unscheduled operations: 10. @@ -74,7 +74,7 @@ Unscheduled operations: 10. | `assurance.build.licence-field` | `tools/ci/tests/integration/topology.rs::a_workspace_package_without_the_apache_licence_is_refused` | performed | | `assurance.build.linked-query-functions` | `crates/contextful-context/tests/integration/read/extension.rs::columnar_reading_and_statement_serialization_are_statically_linked_and_loaded` | performed | | `assurance.build.one-engine-build` | `tools/ci/tests/integration/target_dirs.rs::the_workspace_stage_runs_one_invocation_and_the_store_suites_link_no_engine_without_read` | performed | -| `assurance.build.profile-build` | `tools/ci/tests/integration/features.rs::the_features_stage_tests_each_listed_run_in_parts_one_per_binary_run` | performed | +| `assurance.build.profile-build` | `tools/ci/tests/integration/features.rs::the_features_stage_tests_each_listed_run_in_two_binary_parts` | performed | | `assurance.build.release-artifact` | `tools/ci/tests/integration/artifact.rs::a_dry_run_release_packages_three_archives_with_checksums_sboms_and_formulae` | performed | | `assurance.build.release-builder` | `tools/ci/tests/integration/artifact.rs::zigbuild_packages_a_darwin_cell_and_emits_its_formula_metadata` | performed | | `assurance.build.release-metadata` | `tools/ci/tests/integration/artifact.rs::zigbuild_packages_a_darwin_cell_and_emits_its_formula_metadata` | performed | @@ -139,6 +139,7 @@ Unscheduled operations: 10. | `assurance.gate.stage-sequence` | `tools/ci/tests/integration/stages.rs::the_gate_defines_the_thirteen_stages_in_run_order_and_runs_a_named_subset` | performed | | `assurance.gate.stage-subset` | `tools/ci/tests/integration/stages.rs::a_subset_omitting_a_predecessor_whose_output_is_absent_is_refused_before_any_stage` | performed | | `assurance.gate.surface-check-failed` | `tools/ci/tests/integration/stages.rs::a_failing_surface_check_is_refused_naming_the_surface_and_the_script` | performed | +| `assurance.gate.test-first-parts` | `tools/ci/tests/integration/test_first.rs::a_source_package_requires_its_own_changed_test_when_another_package_is_red` | performed | | `assurance.gate.typescript-surfaces` | `tools/ci/tests/integration/stages.rs::the_surfaces_stage_installs_then_runs_each_declared_check` | performed | | `assurance.gate.workspace-parts` | `tools/ci/tests/integration/target_dirs.rs::remote_workspace_parts_compile_the_union_and_run_each_package_suite` | performed | | `assurance.measure.ledger` | `tools/eval/tests/integration/ledger.rs::an_entry_carries_its_clause_metric_tier_method_and_threshold` | performed | diff --git a/tools/ci/src/main.rs b/tools/ci/src/main.rs index 100f20cca..abd3fe81b 100644 --- a/tools/ci/src/main.rs +++ b/tools/ci/src/main.rs @@ -88,6 +88,12 @@ enum Cmd { /// as its parts. #[arg(long)] parts: bool, + /// Include one test-first part per changed test package against this revision. + #[arg(long, requires = "parts")] + base: Option, + /// Print the selected stage names as a JSON array for a workflow matrix. + #[arg(long)] + json: bool, }, /// Hold every key in a git-tracked `.env*` file to ciphertext under a scope comment. Secrets, @@ -256,11 +262,13 @@ fn refuse(code: &'static str, message: String) -> anyhow::Error { fn main() { let cli = Cli::parse(); let result = match cli.cmd { - Cmd::Stages { parts: false } => { - STAGES.iter().for_each(|s| println!("{s}")); - Ok(()) + Cmd::Stages { parts, base, json } => { + let listed = if parts { repo_root().and_then(|root| dispatched(&root, base.as_deref())) } else { Ok(STAGES.iter().map(|s| s.to_string()).collect()) }; + listed.and_then(|stages| { + if json { println!("{}", serde_json::to_string(&stages)?); } else { stages.iter().for_each(|s| println!("{s}")); } + Ok(()) + }) } - Cmd::Stages { parts: true } => repo_root().and_then(|root| dispatched(&root)).map(|all| all.iter().for_each(|s| println!("{s}"))), Cmd::Gate { stages, predecessors, base, base_bound_secs } => gate(&stages, predecessors, &base, Duration::from_secs(base_bound_secs)), Cmd::Secrets => repo_root().and_then(|root| secrets(&root)), Cmd::Mirrors => repo_root().and_then(|root| mirrors(&root)), @@ -378,18 +386,19 @@ fn repo_root() -> Result { /// The stages whose work splits into parts FlareDispatch dispatches one check /// each, so each part fits one stage's wall clock (`assurance.build.profile-build`, /// `assurance.gate.budget-stage`). -const SPLIT: [&str; 3] = ["workspace", "features", "budget"]; +const SPLIT: [&str; 4] = ["test-first", "workspace", "features", "budget"]; /// The parts of a split `stage`: the features stage's `packages`, every featured package but -/// the binary, then `binary-` per run of the binary package; the budget stage's +/// the binary, then `binary-` and `formal-` per binary feature run; the budget stage's /// `` per profile the binary declares, its name after `contextful-`. -fn parts(root: &Path, stage: &str) -> Result> { +fn parts(root: &Path, stage: &str, base: Option<&str>) -> Result> { Ok(match stage { - "workspace" => ["compile", "cli", "foundation", "runtime", "read", "ci"].into_iter().map(str::to_string).collect(), + "test-first" => match base { Some(base) => test_first_parts(root, base)?, None => vec!["validate".into()] }, + "workspace" => ["compile", "cli", "cli-formal", "foundation", "runtime", "read", "ci"].into_iter().map(str::to_string).collect(), "features" => { let featured = featured_packages(root)?; let binary = featured.iter().filter(|p| p.name == topology::BINARY).flat_map(Featured::runs); - std::iter::once("packages".to_string()).chain(binary.map(|(label, _)| format!("binary-{label}"))).collect() + std::iter::once("packages".to_string()).chain(binary.flat_map(|(label, _)| [format!("binary-{label}"), format!("formal-{label}")])).collect() } "budget" => topology::declared_profiles(root)?.iter().map(|p| p.strip_prefix("contextful-").unwrap_or(p).to_string()).collect(), _ => Vec::new(), @@ -398,11 +407,11 @@ fn parts(root: &Path, stage: &str) -> Result> { /// What FlareDispatch dispatches, in run order: each stage, a split stage as /// `.` per part. -fn dispatched(root: &Path) -> Result> { +fn dispatched(root: &Path, base: Option<&str>) -> Result> { let mut out = Vec::new(); for stage in STAGES { if SPLIT.contains(&stage) { - out.extend(parts(root, stage)?.into_iter().map(|p| format!("{stage}.{p}"))); + out.extend(parts(root, stage, base)?.into_iter().map(|p| format!("{stage}.{p}"))); } else { out.push(stage.to_string()); } @@ -425,10 +434,10 @@ fn gate(named: &[String], predecessors: bool, base: &str, bound: Duration) -> Re stages.push(stage.to_string()); match part { None => whole.push(stage), - Some(part) if SPLIT.contains(&stage) && parts(&root, stage)?.iter().any(|p| p == part) => { + Some(part) if SPLIT.contains(&stage) && parts(&root, stage, Some(base))?.iter().any(|p| p == part) => { narrowed.entry(stage).or_default().push(part.to_string()) } - Some(_) => bail!("no part `{name}`; the parts are {}", dispatched(&root)?.join(", ")), + Some(_) => bail!("no part `{name}`; the parts are {}", dispatched(&root, None)?.join(", ")), } } narrowed.retain(|stage, _| !whole.contains(stage)); @@ -461,7 +470,7 @@ fn run_stage(root: &Path, stage: &str, only: Option<&[String]>, base: &str, boun "test-first" => { provision_lean(root)?; provision_wasm(root)?; - test_first(root, base, bound)? + test_first(root, base, bound, only)? } "workspace" => { provision_lean(root)?; @@ -588,6 +597,8 @@ fn workspace_part(package: &str) -> &'static str { fn workspace(root: &Path, only: Option<&[String]>) -> Result<()> { if let Some(only) = only { + let ordinary = only.iter().any(|part| part == "cli"); + let formal = only.iter().any(|part| part == "cli-formal"); if only.iter().any(|part| part == "compile") { let mut args = vec!["test", "--workspace"]; if root.join(ACCEPTANCE_DIR).join("Cargo.toml").exists() { @@ -598,15 +609,21 @@ fn workspace(root: &Path, only: Option<&[String]>) -> Result<()> { } let packages: Vec = workspace_packages(root)? .into_iter() - .filter(|package| only.iter().any(|part| part == workspace_part(package))) + .filter(|package| only.iter().any(|part| part == workspace_part(package)) || (formal && package == topology::BINARY)) .collect(); - if !packages.is_empty() { + let other: Vec<&String> = packages.iter().filter(|package| package.as_str() != topology::BINARY).collect(); + if !other.is_empty() { let mut args = vec!["test"]; - for package in &packages { + for package in other { args.extend(["--package", package.as_str()]); } run_staged(root, "workspace", &args)?; } + if packages.iter().any(|package| package == topology::BINARY) { + let mut args = vec!["test", "--package", topology::BINARY]; + cli_partition(&mut args, ordinary, formal); + run_staged(root, "workspace", &args)?; + } return Ok(()); } let mut args = vec!["test", "--workspace"]; @@ -616,12 +633,23 @@ fn workspace(root: &Path, only: Option<&[String]>) -> Result<()> { run_staged(root, "workspace", &args) } +/// The differential module runs separately from the other CLI tests in remote parts; +/// selecting both partitions retains Cargo's complete suite, including doctests. +fn cli_partition(args: &mut Vec<&str>, ordinary: bool, formal: bool) { + if !ordinary { + args.extend(["--test", "integration", "differential::"]); + } else if !formal { + args.extend(["--", "--skip", "differential::"]); + } +} + /// The feature combinations the workspace stage's unified build does not reach /// (`assurance.build.staged-feature-runs`): each workspace package declaring a feature other /// than `default` runs its suite alone, once with no features, once with every feature, and /// once per feature set its manifest lists under `[package.metadata.contextful] /// feature-runs` (`assurance.build.profile-build`). `only` narrows the stage to its named -/// parts: `packages`, every package but the binary, and `binary-`, one run of the binary. +/// parts: `packages`, every package but the binary, and `binary-` plus `formal-` +/// for the binary's ordinary and differential tests under each feature set. /// `cargo test -p` resolves the selected package's features without its dependents', so the /// store adapter's write suites run with the read face off /// (`topology.package.store-write-engine-free`) and the policy package's without `exchange`. @@ -637,15 +665,18 @@ fn features(root: &Path, only: Option<&[String]>) -> Result<()> { for package in &featured { let name = &package.name; let binary = name == topology::BINARY; - let selected = |label: &str| { - only.is_none_or(|o| o.iter().any(|p| if binary { p.strip_prefix("binary-") == Some(label) } else { p == "packages" })) - }; - for (_, combination) in package.runs().into_iter().filter(|(label, _)| selected(label)) { + for (label, combination) in package.runs() { + let ordinary = only.is_none_or(|o| o.iter().any(|p| if binary { p.strip_prefix("binary-") == Some(label.as_str()) } else { p == "packages" })); + let formal = binary && only.is_none_or(|o| o.iter().any(|p| p.strip_prefix("formal-") == Some(label.as_str()))); + if !ordinary && !formal { continue; } let shown = combination.join(" "); eprintln!("features: {name} {shown}"); + let mut partition = Vec::new(); + if binary { cli_partition(&mut partition, ordinary, formal); } let status = Command::new("cargo") .args(["test", "-p", name]) .args(&combination) + .args(&partition) .env("CARGO_TARGET_DIR", &target) .current_dir(root) .status()?; @@ -951,21 +982,79 @@ fn workspace_packages(root: &Path) -> Result> { // ---------------------------------------------------------------- test-first -fn test_first(root: &Path, base: &str, bound: Duration) -> Result<()> { +fn changed_tests(base: &str) -> Result> { let range = format!("{base}...HEAD"); + Ok(git(&["diff", "--name-only", "--diff-filter=ACMR", &range])?.lines().filter(|p| is_test(p)).map(str::to_string).collect()) +} + +fn changed_package(root: &Path, path: &str) -> Result { + let manifest = Path::new(path) + .ancestors() + .skip(1) + .map(|dir| root.join(dir).join("Cargo.toml")) + .find(|manifest| manifest.is_file()) + .with_context(|| format!("`{path}` belongs to no Cargo package"))?; + let contents = std::fs::read_to_string(&manifest).with_context(|| format!("reading {}", manifest.display()))?; + let value: toml::Value = contents.parse().with_context(|| format!("parsing {}", manifest.display()))?; + Ok(value.get("package").and_then(|p| p.get("name")).and_then(toml::Value::as_str).context("the changed path's package has a name")?.to_string()) +} + +fn test_first_parts(root: &Path, base: &str) -> Result> { + let mut packages = std::collections::BTreeSet::new(); + for source in sources_outside_refactors(base)? { + packages.insert(changed_package(root, &source)?); + } + if !packages.is_empty() { + for test in changed_tests(base)? { + packages.insert(changed_package(root, &test)?); + } + } + Ok(std::iter::once("validate".to_string()).chain(packages).collect()) +} + +fn test_first(root: &Path, base: &str, bound: Duration, only: Option<&[String]>) -> Result<()> { let sources = sources_outside_refactors(base)?; if sources.is_empty() { eprintln!("test-first: no Rust source under crates/ or tools/ changed outside `Test-First: {REFACTOR_TRAILER}` commits"); return Ok(()); } - let added = git(&["diff", "--name-only", "--diff-filter=ACMR", &range])?; - let tests: Vec<&str> = added.lines().filter(|p| is_test(p)).collect(); + let changed = changed_tests(base)?; + let tests: Vec<&str> = changed.iter().map(String::as_str).collect(); if tests.is_empty() { return Err(refuse( "TestNotFirst", format!("{} source file(s) changed and no test under a package's tests/ did: {}", sources.len(), sources.join(", ")), )); } + let source_packages: std::collections::BTreeSet = sources.iter().map(|source| changed_package(root, source)).collect::>()?; + let test_packages: std::collections::BTreeSet = tests.iter().map(|test| changed_package(root, test)).collect::>()?; + if only.is_none() || only.is_some_and(|parts| parts == ["validate"]) { + if let Some(missing) = source_packages.difference(&test_packages).next() { + return Err(refuse("TestNotFirst", format!("source package `{missing}` has no changed test under its tests/"))); + } + } + let selected: Vec<&str> = match only { + Some(parts) if parts.len() == 1 && parts[0] == "validate" => return Ok(()), + Some(parts) => { + let mut selected = Vec::new(); + for test in tests { + if parts.contains(&changed_package(root, test)?) { + selected.push(test); + } + } + selected + } + None => tests, + }; + let mut by_package: std::collections::BTreeMap> = Default::default(); + for test in selected { + by_package.entry(changed_package(root, test)?).or_default().push(test); + } + if let Some(parts) = only { + if let Some(missing) = parts.iter().filter(|part| part.as_str() != "validate").find(|part| !by_package.contains_key(part.as_str())) { + return Err(refuse("TestNotFirst", format!("source package `{missing}` has no changed test under its tests/"))); + } + } let scratch = stage_target(root, "test-first"); let tree = scratch.join("tree"); @@ -973,16 +1062,20 @@ fn test_first(root: &Path, base: &str, bound: Duration) -> Result<()> { let _ = std::fs::remove_dir_all(&scratch); std::fs::create_dir_all(&scratch)?; git(&["worktree", "add", "--detach", "-q", &tree.to_string_lossy(), &merge_base(base)?])?; - let verdict = red_against_base(root, &tree, &scratch.join("target"), &tests, bound); + let verdict = (|| -> Result> { + let mut all_red = Vec::new(); + for (package, tests) in by_package { + let red = red_against_base(root, &tree, &scratch.join("target"), &tests, bound)?; + if red.is_empty() { + return Err(refuse("TestNotFirst", format!("the change's `{package}` tests pass against the base source, so they specify nothing it adds: {}", tests.join(", ")))); + } + all_red.extend(red); + } + Ok(all_red) + })(); let _ = git(&["worktree", "remove", "--force", &tree.to_string_lossy()]); let _ = std::fs::remove_dir_all(&scratch); let red = verdict?; - if red.is_empty() { - return Err(refuse( - "TestNotFirst", - format!("the change's tests pass against the base source, so they specify nothing it adds: {}", tests.join(", ")), - )); - } eprintln!("test-first: red against the base in {}", red.join(", ")); Ok(()) } @@ -1211,7 +1304,11 @@ fn red_against_base(root: &Path, tree: &Path, target: &Path, tests: &[&str], bou if !matches!(listed, BaseRun::Passed) { eprint!("{}{}", printed.stdout, printed.stderr); unrunnable(&printed)?; - red.push(format!("{label}, whose tests do not list at base")); + red.push(match listed { + BaseRun::Killed => format!("{label}, listing killed at the {} s bound", bound.as_secs()), + BaseRun::Failed => format!("{label}, whose tests do not list at base"), + BaseRun::Passed => unreachable!("a passing listing has no red verdict"), + }); continue; } let names: Vec = printed diff --git a/tools/ci/tests/integration/cli_parts.rs b/tools/ci/tests/integration/cli_parts.rs new file mode 100644 index 000000000..479fee1bb --- /dev/null +++ b/tools/ci/tests/integration/cli_parts.rs @@ -0,0 +1,31 @@ +use crate::{manifest, stderr, Repo}; + +#[test] +fn remote_cli_parts_run_every_test_once_and_propagate_differential_failures() { + let r = Repo::init(); + r.write("crates/contextful-cli/Cargo.toml", &(manifest("contextful-cli", "") + "\n[features]\nextra = []\n")); + r.write("crates/contextful-cli/src/lib.rs", ""); + r.write("crates/contextful-cli/tests/integration/main.rs", "#[test] fn ordinary() { std::fs::write(\"ordinary.ran\", \"yes\").unwrap(); }\nmod differential { #[test] fn reference() { std::fs::write(\"reference.ran\", \"yes\").unwrap(); } }\n"); + r.lock(); + r.commit("CLI ordinary and differential suites"); + let package = r.root.join("crates/contextful-cli"); + for (ordinary, differential) in [("workspace.cli", "workspace.cli-formal"), ("features.binary-all", "features.formal-all")] { + let out = r.gate(&["--stage", ordinary]); + assert!(out.status.success(), "{ordinary}: {}", stderr(&out)); + assert!(package.join("ordinary.ran").exists(), "{ordinary} missed ordinary tests"); + assert!(!package.join("reference.ran").exists(), "{ordinary} also ran the differential partition"); + std::fs::remove_file(package.join("ordinary.ran")).unwrap(); + let out = r.gate(&["--stage", differential]); + assert!(out.status.success(), "{differential}: {}", stderr(&out)); + assert!(package.join("reference.ran").exists(), "{differential} missed reference tests"); + assert!(!package.join("ordinary.ran").exists(), "{differential} repeated ordinary tests"); + std::fs::remove_file(package.join("reference.ran")).unwrap(); + } + r.write("crates/contextful-cli/tests/integration/main.rs", "mod differential { #[test] fn reference() { panic!(\"reference regression\"); } }\n"); + r.commit("a failing differential case"); + for part in ["workspace.cli-formal", "features.formal-all"] { + let out = r.gate(&["--stage", part]); + assert!(!out.status.success(), "{part} hid a differential failure"); + assert!(String::from_utf8_lossy(&out.stdout).contains("reference regression"), "{part}: {}", stderr(&out)); + } +} diff --git a/tools/ci/tests/integration/features.rs b/tools/ci/tests/integration/features.rs index 94b1f275a..e30429f47 100644 --- a/tools/ci/tests/integration/features.rs +++ b/tools/ci/tests/integration/features.rs @@ -119,10 +119,10 @@ fn bundled(r: &Repo, lib_edge: &str, runs: &str) { r.write("crates/gated/tests/integration/main.rs", "#[test]\nfn builds() {}\n"); } -/// The features stage tests each package under every feature set its manifest lists in `feature-runs`, in parts dispatched as one check each: the other packages, and each binary run — none, all and each listed set. -// spec: assurance.build.profile-build@5854cf3a +/// Each binary feature run partitions differential tests from its remaining suite. +// spec: assurance.build.profile-build@0250142c #[test] -fn the_features_stage_tests_each_listed_run_in_parts_one_per_binary_run() { +fn the_features_stage_tests_each_listed_run_in_two_binary_parts() { let r = Repo::init(); bundled(&r, "pub fn edge() {}", "\"edge\""); r.write( @@ -140,6 +140,9 @@ fn the_features_stage_tests_each_listed_run_in_parts_one_per_binary_run() { assert!(at("evaluate") < at("features.packages") && at("features.packages") < at("features.binary-none"), "{listed}"); assert!(at("features.binary-none") < at("features.binary-all") && at("features.binary-all") < at("features.binary-extra"), "{listed}"); assert!(at("features.binary-extra") < at("crate-graph") && !parts.contains(&"features"), "{listed}"); + for label in ["none", "all", "extra"] { + assert!(at(&format!("features.binary-{label}")) < at(&format!("features.formal-{label}")), "{listed}"); + } // The whole stage runs every part. let o = features(&r); diff --git a/tools/ci/tests/integration/main.rs b/tools/ci/tests/integration/main.rs index 236d5092b..c6c6697e0 100644 --- a/tools/ci/tests/integration/main.rs +++ b/tools/ci/tests/integration/main.rs @@ -4,6 +4,7 @@ mod acceptance_surface; mod allowlist; mod artifact; +mod cli_parts; mod dependency_versions; mod deny; mod disk; diff --git a/tools/ci/tests/integration/target_dirs.rs b/tools/ci/tests/integration/target_dirs.rs index d3f49962e..1a64e2d0c 100644 --- a/tools/ci/tests/integration/target_dirs.rs +++ b/tools/ci/tests/integration/target_dirs.rs @@ -123,8 +123,8 @@ fn the_workspace_stage_runs_one_invocation_and_the_store_suites_link_no_engine_w } } -/// Remote workspace checks compile the feature-unified workspace separately from the CLI suite and run every other non-acceptance package suite in one of four groups. -// spec: assurance.gate.workspace-parts@2b8d0b2a +/// Workspace compilation and both CLI partitions run separately; four groups run every other package. +// spec: assurance.gate.workspace-parts@4ac089c3 #[test] fn remote_workspace_parts_compile_the_union_and_run_each_package_suite() { let r = Repo::init(); @@ -149,8 +149,16 @@ fn remote_workspace_parts_compile_the_union_and_run_each_package_suite() { assert!(tested.status.success(), "{}", stderr(&tested)); let calls = std::fs::read_to_string(&log).unwrap(); assert!(calls.contains("test --package contextful-cli"), "{calls}"); + assert!(calls.contains("--skip differential::"), "{calls}"); assert_eq!(calls.lines().count(), 1, "CLI runs no workspace compile: {calls}"); + std::fs::remove_file(&log).unwrap(); + let formal = r.gate_with_cargo(&recording(&log, 0), &["--stage", "workspace.cli-formal"]); + assert!(formal.status.success(), "{}", stderr(&formal)); + let calls = std::fs::read_to_string(&log).unwrap(); + assert!(calls.contains("test --package contextful-cli --test integration differential::"), "{calls}"); + assert_eq!(calls.lines().count(), 1, "formal runs no workspace compile: {calls}"); + std::fs::remove_file(&log).unwrap(); let tested = r.gate_with_cargo(&recording(&log, 0), &["--stage", "workspace.foundation"]); assert!(tested.status.success(), "{}", stderr(&tested)); diff --git a/tools/ci/tests/integration/test_first.rs b/tools/ci/tests/integration/test_first.rs index 45060788a..5b2f58254 100644 --- a/tools/ci/tests/integration/test_first.rs +++ b/tools/ci/tests/integration/test_first.rs @@ -68,6 +68,70 @@ fn a_test_failing_on_the_base_passes() { assert!(o.status.success(), "{}", stderr(&o)); } +#[test] +fn each_changed_test_package_has_its_own_dispatch_part() { + let r = Repo::init(); + r.write("crates/other/Cargo.toml", &manifest("other", "")); + r.write("crates/other/src/lib.rs", "pub fn value() -> i32 { 1 }\n"); + r.write("crates/other/tests/integration/main.rs", "mod value;\n"); + r.write("crates/other/tests/integration/value.rs", "#[test]\nfn positive() { assert!(other::value() > 0); }\n"); + r.lock(); + r.commit("second package and workspace lock"); + let base = r.head(); + r.write("crates/demo/src/lib.rs", TRIPLE); + r.write("crates/demo/tests/integration/main.rs", "mod double;\nmod triple;\n"); + r.write("crates/demo/tests/integration/triple.rs", "#[test]\nfn triples() { assert_eq!(demo::triple(2), 6); }\n"); + r.write("crates/other/src/lib.rs", "pub fn value() -> i32 { 2 }\n"); + r.write("crates/other/tests/integration/value.rs", "#[test]\nfn positive() { assert!(other::value() >= 1); }\n"); + r.commit("triple, test first"); + + let listed = r.run_ci(&["stages", "--parts", "--base", &base, "--json"]); + assert!(listed.status.success(), "{}", stderr(&listed)); + let stages: Vec = serde_json::from_slice(&listed.stdout).unwrap(); + assert!(stages.contains(&"test-first.validate".to_string()), "{stages:?}"); + assert!(stages.contains(&"test-first.demo".to_string()), "{stages:?}"); + assert!(stages.contains(&"test-first.other".to_string()), "{stages:?}"); + assert!(!stages.contains(&"test-first".to_string()), "{stages:?}"); + + let part = r.gate(&["--stage", "test-first.demo", "--base", &base]); + assert!(part.status.success(), "{}", stderr(&part)); + let part = r.gate(&["--stage", "test-first.other", "--base", &base]); + assert!(!part.status.success() && stderr(&part).contains("TestNotFirst"), "{}", stderr(&part)); + let whole = r.gate(&["--stage", "test-first", "--base", &base]); + assert!(!whole.status.success() && stderr(&whole).contains("other"), "{}", stderr(&whole)); + let validation = r.gate(&["--stage", "test-first.validate", "--base", &base]); + assert!(validation.status.success(), "{}", stderr(&validation)); +} + +// spec: assurance.gate.test-first-parts@0bd45e25 +#[test] +fn a_source_package_requires_its_own_changed_test_when_another_package_is_red() { + let r = Repo::init(); + r.write("crates/other/Cargo.toml", &manifest("other", "")); + r.write("crates/other/src/lib.rs", "pub fn value() -> i32 { 1 }\n"); + r.write("crates/other/tests/integration/main.rs", "mod value;\n"); + r.write("crates/other/tests/integration/value.rs", "#[test]\nfn value() { assert_eq!(other::value(), 1); }\n"); + r.lock(); + r.commit("second package and workspace lock"); + let base = r.head(); + r.write("crates/demo/src/lib.rs", TRIPLE); + r.write("crates/other/src/lib.rs", "pub fn value() -> i32 { 2 }\n"); + r.write("crates/other/tests/integration/value.rs", "#[test]\nfn value() { assert_eq!(other::value(), 2); }\n"); + r.commit("two source packages, one changed test package"); + + let listed = r.run_ci(&["stages", "--parts", "--base", &base, "--json"]); + assert!(listed.status.success(), "{}", stderr(&listed)); + let stages: Vec = serde_json::from_slice(&listed.stdout).unwrap(); + assert!(stages.contains(&"test-first.demo".to_string()), "{stages:?}"); + assert!(stages.contains(&"test-first.other".to_string()), "{stages:?}"); + + let validation = r.gate(&["--stage", "test-first.validate", "--base", &base]); + assert!(!validation.status.success() && stderr(&validation).contains("TestNotFirst"), "{}", stderr(&validation)); + assert!(stderr(&validation).contains("demo"), "{}", stderr(&validation)); + let whole = r.gate(&["--stage", "test-first", "--base", &base]); + assert!(!whole.status.success() && stderr(&whole).contains("demo"), "{}", stderr(&whole)); +} + #[test] fn a_refactor_trailer_exempts_the_range() { let r = Repo::init(); @@ -92,21 +156,22 @@ fn a_refactor_commit_does_not_exempt_the_rest_of_the_range() { } #[test] -fn a_new_package_counts_red_alone_and_leaves_the_base_workspace_loadable() { +fn a_new_package_counts_red_and_leaves_the_base_workspace_loadable() { let r = Repo::init(); let base = r.head(); - // A new package with its own tests, beside a green-on-base test for a source change in `demo`. + // Both changed packages provide tests red against their own base source. r.write("crates/fresh/Cargo.toml", &crate::manifest("fresh", "")); r.write("crates/fresh/src/lib.rs", "pub fn one() -> i32 {\n 1\n}\n"); r.write("crates/fresh/tests/integration/main.rs", "#[test]\nfn one() {\n assert_eq!(fresh::one(), 1);\n}\n"); r.write("crates/demo/src/lib.rs", TRIPLE); - r.write("crates/demo/tests/integration/double.rs", "#[test]\nfn doubles() {\n assert_eq!(demo::double(3), 6);\n}\n"); - r.commit("a new package, and a demo test that specifies nothing new"); + r.write("crates/demo/tests/integration/main.rs", "mod double;\nmod triple;\n"); + r.write("crates/demo/tests/integration/triple.rs", "#[test]\nfn triples() { assert_eq!(demo::triple(2), 6); }\n"); + r.commit("two packages with red tests"); let o = r.gate(&["--stage", "test-first", "--base", &base]); let err = stderr(&o); assert!(o.status.success(), "{err}"); assert!(err.contains("crates/fresh (absent at base)"), "{err}"); - assert!(!err.contains("crates/demo"), "demo's test passes at base and counts green: {err}"); + assert!(err.contains("crates/demo"), "demo's test must count red: {err}"); assert!(!err.contains("failed to load manifest"), "{err}"); } @@ -207,6 +272,20 @@ fn a_test_quoting_an_infrastructure_fault_still_reads_as_red() { assert!(err.contains("red against the base in crates/demo"), "{err}"); } +#[test] +fn a_base_listing_past_its_bound_reports_the_package_and_bound() { + let r = Repo::init(); + let base = r.head(); + r.write("crates/demo/src/lib.rs", TRIPLE); + r.write("crates/demo/tests/integration/double.rs", "#[test]\nfn doubles() {\n assert_eq!(demo::double(4), 8);\n}\n"); + r.commit("triple, with a changed double test"); + let cargo = "case \" $* \" in\n *\" --list \"*) sleep 30 ;;\nesac\n"; + let o = r.gate_with_cargo(cargo, &["--stage", "test-first", "--base", &base, "--base-bound-secs", "20"]); + let err = stderr(&o); + assert!(o.status.success(), "{err}"); + assert!(err.contains("crates/demo") && err.contains("listing killed at the 20 s bound"), "{err}"); +} + #[test] fn a_slow_base_build_is_not_charged_to_the_bound() { let r = Repo::init(); diff --git a/tools/ci/tests/integration/workflow.rs b/tools/ci/tests/integration/workflow.rs index 2ef0fe148..9c032888d 100644 --- a/tools/ci/tests/integration/workflow.rs +++ b/tools/ci/tests/integration/workflow.rs @@ -17,7 +17,7 @@ fn every_gate_stage_has_a_dispatchable_part() { .collect(); assert_eq!( parts.len(), - 24, + 29, "the remote gate expects one check per part: {parts:?}" ); assert!(parts.iter().any(|part| part == "workspace.cli"), "the CLI suite has no separate remote check: {parts:?}"); From 49261cb8cb2548963aff1f37497e67ce9fffb7ad Mon Sep 17 00:00:00 2001 From: debuggingfuture Date: Fri, 9 Oct 2026 00:59:33 +0800 Subject: [PATCH 2/3] fix(ci): dispatch test-first parts for changed source packages only A package whose tests change without source change holds no red obligation, and validation runs whenever its part is selected beside package parts. --- AGENTS.md | 2 +- spec/81-engineering.md | 2 +- spec/spec.lock.json | 2 +- tools/ci/src/main.rs | 13 ++----- tools/ci/tests/integration/test_first.rs | 47 +++++++++++++++++++++++- 5 files changed, 53 insertions(+), 13 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index ec682d94a..dcce316bd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -122,7 +122,7 @@ The gate measures commits, so commit before running it. The FlareDispatch GitHub App dispatches `contextful-ci gate` from same-repository pull-request heads. Its `contextful-gate` run publishes the `flare-dispatch/contextful-gate` parent, 29 static child check-runs, and one -test-first child per changed package: +test-first child per changed source package: `flare-dispatch/check:` for each of `pins`, `toolchain`, `schema`, `workspace`, `acceptance`, `evaluate`, `features`, `crate-graph`, `connectors`, `surfaces`, `formal` and `budget`. The test-first, features and budget stages dispatch one check per diff --git a/spec/81-engineering.md b/spec/81-engineering.md index 1bfad4edd..6134e6f91 100644 --- a/spec/81-engineering.md +++ b/spec/81-engineering.md @@ -188,7 +188,7 @@ Stage order, secrets of record, the crate-graph, row-token, egress and dependenc *A-assurance* - `windows-checks` — Windows native checks run only as optional diagnostics and contribute no required merge or release verdict. *A-assurance* -- `test-first-parts` — Remote validation requires a changed test in each changed source package under {{assurance.test.test-first}}; each changed source or test package checks only its own tests against the base in a separate dispatch. +- `test-first-parts` — Remote validation requires a changed test in each changed source package under {{assurance.test.test-first}}; each changed source package checks only its own changed tests against the base in a separate dispatch. *because one base build per package fits the sandbox's wall clock more reliably* - `workspace-parts` — Remote workspace checks compile the feature-unified workspace without running tests, run differential and remaining CLI tests separately, and run each other non-acceptance package suite in exactly one of four groups. *A-assurance* diff --git a/spec/spec.lock.json b/spec/spec.lock.json index d7c5e59c5..e72166fc3 100644 --- a/spec/spec.lock.json +++ b/spec/spec.lock.json @@ -22281,7 +22281,7 @@ "operation": "gate", "subject": "test-first-parts", "kind": "behavior", - "statement": "Remote validation requires a changed test in each changed source package under {{assurance.test.test-first}}; each changed source or test package checks only its own tests against the base in a separate dispatch.", + "statement": "Remote validation requires a changed test in each changed source package under {{assurance.test.test-first}}; each changed source package checks only its own changed tests against the base in a separate dispatch.", "why": "because one base build per package fits the sandbox's wall clock more reliably", "file": "spec/81-engineering.md", "line": 191 diff --git a/tools/ci/src/main.rs b/tools/ci/src/main.rs index abd3fe81b..4947e0587 100644 --- a/tools/ci/src/main.rs +++ b/tools/ci/src/main.rs @@ -1004,11 +1004,6 @@ fn test_first_parts(root: &Path, base: &str) -> Result> { for source in sources_outside_refactors(base)? { packages.insert(changed_package(root, &source)?); } - if !packages.is_empty() { - for test in changed_tests(base)? { - packages.insert(changed_package(root, &test)?); - } - } Ok(std::iter::once("validate".to_string()).chain(packages).collect()) } @@ -1028,23 +1023,23 @@ fn test_first(root: &Path, base: &str, bound: Duration, only: Option<&[String]>) } let source_packages: std::collections::BTreeSet = sources.iter().map(|source| changed_package(root, source)).collect::>()?; let test_packages: std::collections::BTreeSet = tests.iter().map(|test| changed_package(root, test)).collect::>()?; - if only.is_none() || only.is_some_and(|parts| parts == ["validate"]) { + if only.is_none_or(|parts| parts.iter().any(|part| part == "validate")) { if let Some(missing) = source_packages.difference(&test_packages).next() { return Err(refuse("TestNotFirst", format!("source package `{missing}` has no changed test under its tests/"))); } } let selected: Vec<&str> = match only { Some(parts) if parts.len() == 1 && parts[0] == "validate" => return Ok(()), - Some(parts) => { + only => { let mut selected = Vec::new(); for test in tests { - if parts.contains(&changed_package(root, test)?) { + let package = changed_package(root, test)?; + if source_packages.contains(&package) && only.is_none_or(|parts| parts.contains(&package)) { selected.push(test); } } selected } - None => tests, }; let mut by_package: std::collections::BTreeMap> = Default::default(); for test in selected { diff --git a/tools/ci/tests/integration/test_first.rs b/tools/ci/tests/integration/test_first.rs index 5b2f58254..771449ec8 100644 --- a/tools/ci/tests/integration/test_first.rs +++ b/tools/ci/tests/integration/test_first.rs @@ -103,7 +103,7 @@ fn each_changed_test_package_has_its_own_dispatch_part() { assert!(validation.status.success(), "{}", stderr(&validation)); } -// spec: assurance.gate.test-first-parts@0bd45e25 +// spec: assurance.gate.test-first-parts@183cbf13 #[test] fn a_source_package_requires_its_own_changed_test_when_another_package_is_red() { let r = Repo::init(); @@ -132,6 +132,51 @@ fn a_source_package_requires_its_own_changed_test_when_another_package_is_red() assert!(!whole.status.success() && stderr(&whole).contains("demo"), "{}", stderr(&whole)); } +#[test] +fn a_test_package_without_changed_source_is_neither_dispatched_nor_held_red() { + let r = Repo::init(); + r.write("crates/other/Cargo.toml", &manifest("other", "")); + r.write("crates/other/src/lib.rs", "pub fn value() -> i32 { 1 }\n"); + r.write("crates/other/tests/integration/main.rs", "mod value;\n"); + r.write("crates/other/tests/integration/value.rs", "#[test]\nfn value() { assert_eq!(other::value(), 1); }\n"); + r.lock(); + r.commit("second package and workspace lock"); + let base = r.head(); + r.write("crates/demo/src/lib.rs", TRIPLE); + r.write("crates/demo/tests/integration/main.rs", "mod double;\nmod triple;\n"); + r.write("crates/demo/tests/integration/triple.rs", "#[test]\nfn triples() { assert_eq!(demo::triple(2), 6); }\n"); + r.write("crates/other/tests/integration/value.rs", "#[test]\nfn value_is_one() { assert_eq!(other::value(), 1); }\n"); + r.commit("triple, test first, and a renamed test in another package"); + + let listed = r.run_ci(&["stages", "--parts", "--base", &base, "--json"]); + assert!(listed.status.success(), "{}", stderr(&listed)); + let stages: Vec = serde_json::from_slice(&listed.stdout).unwrap(); + assert!(stages.contains(&"test-first.demo".to_string()), "{stages:?}"); + assert!(!stages.contains(&"test-first.other".to_string()), "{stages:?}"); + + let whole = r.gate(&["--stage", "test-first", "--base", &base]); + assert!(whole.status.success(), "{}", stderr(&whole)); +} + +#[test] +fn validation_runs_beside_selected_package_parts() { + let r = Repo::init(); + r.write("crates/other/Cargo.toml", &manifest("other", "")); + r.write("crates/other/src/lib.rs", "pub fn value() -> i32 { 1 }\n"); + r.write("crates/other/tests/integration/main.rs", "mod value;\n"); + r.write("crates/other/tests/integration/value.rs", "#[test]\nfn value() { assert_eq!(other::value(), 1); }\n"); + r.lock(); + r.commit("second package and workspace lock"); + let base = r.head(); + r.write("crates/demo/src/lib.rs", TRIPLE); + r.write("crates/other/src/lib.rs", "pub fn value() -> i32 { 2 }\n"); + r.write("crates/other/tests/integration/value.rs", "#[test]\nfn value() { assert_eq!(other::value(), 2); }\n"); + r.commit("two source packages, one changed test package"); + + let o = r.gate(&["--stage", "test-first.validate", "--stage", "test-first.other", "--base", &base]); + assert!(!o.status.success() && stderr(&o).contains("source package `demo`"), "{}", stderr(&o)); +} + #[test] fn a_refactor_trailer_exempts_the_range() { let r = Repo::init(); From ae77d2bcf1c135e1236f3b1ca12a6d52c9c57564 Mon Sep 17 00:00:00 2001 From: debuggingfuture Date: Fri, 9 Oct 2026 01:01:14 +0800 Subject: [PATCH 3/3] docs(ci): count 29 static ruleset contexts --- .github/rulesets/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/rulesets/README.md b/.github/rulesets/README.md index 49bf015d7..39ce4e7f2 100644 --- a/.github/rulesets/README.md +++ b/.github/rulesets/README.md @@ -1,7 +1,7 @@ # FlareDispatch required checks -`contextful-gate.proposed.json` is a disabled repository ruleset for `main`. It requires the `flare-dispatch/contextful-gate` parent and the 24 child contexts returned by `contextful-ci stages --parts`; the parent covers fanout and join failures, and a missing child blocks merge. +`contextful-gate.proposed.json` is a disabled repository ruleset for `main`. It requires the `flare-dispatch/contextful-gate` parent and the 29 static child contexts returned by `contextful-ci stages --parts`; per-package `test-first.` children vary by change and stay outside the static list; the parent covers fanout and join failures, and a missing child blocks merge. The list is static while the CLI stage list is derived from the repository. A change to `contextful-ci stages --parts` requires the same change to the proposed ruleset before enforcement. The corpus states the remote obligation in `assurance.gate.remote-check`; local pins prove only CLI behavior, while FlareDispatch owns the webhook and check-run implementation. -The proposal carries `enforcement: disabled` and no `integration_id`. [GitHub's required-check rules](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/available-rules-for-rulesets#require-status-checks-to-pass-before-merging) admit a specific GitHub App as the expected source after its checks exist. The active ruleset requires all 24 contexts to report terminal conclusions on a fresh pull-request head and the operator's App ID in each required context; that tenant-specific ID stays outside the repository. +The proposal carries `enforcement: disabled` and no `integration_id`. [GitHub's required-check rules](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/available-rules-for-rulesets#require-status-checks-to-pass-before-merging) admit a specific GitHub App as the expected source after its checks exist. The active ruleset requires all 29 static contexts to report terminal conclusions on a fresh pull-request head and the operator's App ID in each required context; that tenant-specific ID stays outside the repository.