[Cedric Vidal (cedricvidal)]

Summary

Complete technical delivery of the approved isolated Taxonome spike: a standalone TypeScript API and polished CLI driven by exact scope/taxonomy/v1 YAML; prompt-based Skipped Invocation through a real Ax typed program and GitHub Copilot SDK; deterministic Timeout; safe plain-ATIF and existing Scope .tar.gz archive ingestion into one extra.scope stream; explicit local/package deterministic-handler plugins; bounded optimization/save/restore/replay; and deterministic behavior/source readouts.

The frozen model-experiment source/evidence head remains 7d217d727ac0adaa4cc4121767316fab62d09f3c. The current implementation/evidence head is 9f4d5cd0f78244bb2d075c98275705edf6c05940, based on the prepared PR #1517 parent 6ae17ce30247bb893ab85af170c4c5e290228dd5 and published in native stack 1534. Historical prompt contracts, fixtures/gold, model outputs, complete Ax artifacts, and the original byte-identical 256-call snapshot remain preserved. The first current-YAML demo is retained as explicitly ID-confounded historical evidence. A separately versioned opaque-ID control adds 11 further fully accounted calls on the same hard-500 ledger without changing either earlier packet.

The Taxonome runtime remains portable without Scope API, Judge, MongoDB, Redis, queues, or Experiment APIs; stack placement does not introduce a runtime dependency. The generic runtime is now taxonomy-engine; the shipped agent-behavior YAML, fixtures, and scope/timeout-v1 handler are isolated in minimal ax-taxonomy. The CLI explicitly composes both packages, and the engine has no catalog dependency. Keep draft: human acceptance/Phase 1 exit, production adoption, merge, and later phases remain unauthorized.

Finding: retain the simplified occurrence model for review and prefer the revised baseline prompt for this packet. Actual Ax optimization selected unchanged instructions plus three learned traces; it did not improve final accuracy and increased prompt input. Confidence/output-category duplication was removed; explicit completeness/applicability belongs in the runtime, not model guesswork. Earlier genuine model failures are preserved, not relabeled.

Demos

Before

N/A: new isolated experimental CLI; no previous Taxonome command.

After

terminal.mp4

Comprehensive 21-beat walkthrough using the real taxonome executable. It covers typed deterministic output contracts, plugins, plain/archive provenance, the custom prompt element, an opaque-ID corpus verified against the production prompt projection, a fresh neutral prompt run, held-out eval, one bounded optimize, artifact replay, the full shipped taxonomy, readout/API, and cumulative accounting. The original 256-call snapshot and the first ID-confounded 11-call demo remain immutable; the neutral control adds 11 successful calls for 278/500 total. Neutral baseline and replay both pass 2/2, Ax selects three learned traces but makes 0 instruction proposals, and no accuracy gain is claimed.

Testing

Exact command	Result
pnpm install --offline --frozen-lockfile	Passed: lockfile current
pnpm --filter shared build	Passed
pnpm --filter taxonomy-engine build	Passed
pnpm --filter ax-taxonomy build	Passed
pnpm --filter taxonome build	Passed
pnpm --filter taxonome-custom-handler-example build	Passed: example remains fully type-checked separately from process-test fixture emission
pnpm --filter taxonomy-engine lint	Passed, TypeScript no-emit check
pnpm --filter ax-taxonomy lint	Passed, TypeScript no-emit check
pnpm --filter taxonome lint	Passed, TypeScript no-emit check
cd packages/taxonomy-engine && pnpm test	Passed: 44 tests / 13 files; generic neutral fixtures only
cd packages/ax-taxonomy && pnpm test	Passed: 18 tests / 2 files; focused catalog definition and Timeout tests only
pnpm exec vitest run packages/taxonomy-engine/src packages/ax-taxonomy/src apps/taxonome/src --reporter=default	Passed: 76 tests / 18 files
pnpm exec vitest run apps/taxonome/src/cli.test.ts --reporter=default with both taxonomy-engine and ax-taxonomy dist/ directories absent	Passed: 7/7; trusted fixture emitted to an isolated test-owned directory and cleaned
pnpm --filter static-prompt-evals test	Passed: 38 TypeScript / 94 Python tests
pnpm --filter static-prompt-evals typecheck	Passed
pnpm --filter static-prompt-evals validate-data	Passed: 226 cases, 19 quality targets + 1 standalone SDK target
pnpm notice:check	Passed; NOTICE and NOTICE-REVIEW.txt current
pnpm headers:check	Passed: all 1107 in-scope files
taxonome run inputs/built-in.atif.json --taxonomy taxonomies/built-in/taxonomy.yaml --ledger "$DEMO_LEDGER"	Passed using the direct executable: shipped taxonomy ran prompt-based Skipped Invocation and deterministic Timeout
cd examples/taxonome-custom-handler && taxonome run inputs/run.atif.json --taxonomy taxonomy.yaml --plugin ./dist/handlers/index.js --elements acme-retry-loop	Passed: positive custom-handler result with provider integrity provenance
cd examples/taxonome-custom-handler && taxonome run inputs/run.tar.gz --taxonomy taxonomy.yaml --plugin ./dist/handlers/index.js --elements acme-retry-loop	Passed: semantically equivalent result plus preserved archive origin
pnpm exec vitest run packages/taxonomy-engine/src/evidence.test.ts --reporter=default	Passed: frozen historical artifacts and all three separate 11-call deltas match committed hashes

The focused suite covers YAML normalization/digests/references and schema limits; explicit local/package plugins, reserved/duplicate IDs, clean-CI fixture isolation, and cwd resolution; archive traversal/link/device/duplicate/root/size/count/depth protection, completeness and cross-iteration call references; extra.scope grounding and configuration-versus-availability; immutable deterministic inputs; one-call prompt execution; v2 artifact compatibility; readout distinct-run/occurrence counts; TTY/pipe/NO_COLOR behavior; bounded HTTP retries; ledger concurrency; and frozen evidence hashes. Source/artifact sanitization checks passed. No credentials, host paths, or original requirement snapshots are committed.

Historical experiment-head GitHub CI: run 39592980292, successful and pinned to 7d217d727ac0adaa4cc4121767316fab62d09f3c. Current review-head checks, including documentation-only follow-ups, are shown live on this draft PR.

Actual SDK evidence (not mocks)

First YAML prompt demo (historical, ID-confounded): the initial 11-call packet is preserved byte-for-byte, but its model-visible session IDs contained descriptive positive/negative hints. Its 2/2 result is not treated as a clean prompt-quality control, and its outputs are not relabeled.

Opaque-ID control (scope/taxonomy/v1): a deterministic regression serializes every case through the actual production taxonomyPromptEvidence() projection. Case/family/origin metadata is absent; all forwarded session/run/scenario/profile identities are opaque; transcript semantics and gold are unchanged.

• taxonome run inputs/repeated-request.atif.json ... — completed positive using opaque run run-9f31 with grounded turns 1 and 2.
• taxonome eval ... --cases cases/neutral-prompt-cases.jsonl --gold cases/neutral-prompt-gold.json --split heldout ... — neutral baseline 2/2 using 2 calls on case-05 and case-06.
• taxonome optimize ... --max-metric-calls 12 ... — one bounded Ax run using 5 calls (3 teacher, 2 student), selecting three learned traces and two examples across five metric entries. The artifact records 0 instruction proposals and earlyStopped:false; no instruction rewrite or early termination is claimed.
• Re-evaluation with the neutral artifact — 2/2 using 2 calls; no held-out accuracy gain.
• Full shipped taxonomy run — Skipped Invocation prompt observed false; deterministic Timeout observed true.

Neutral-control accounting: 11 successful calls, 13691 input / 252 output tokens, 11 observed model attempts, no failures, USD cost unknown. Cumulative ledger: 278/500, 222 remaining. The original 256-call snapshot and first 11-call delta remain unchanged; the neutral delta and outputs live under evidence/yaml-prompt-neutral-control/.

Historical v1 experiment (preserved snapshot):

The commands below are the historical v1 command shapes from the frozen experiment head, retained to explain the archived evidence; they are not replay instructions for the current YAML CLI. The current CLI requires --taxonomy, and old taxonome.ax.v1 artifacts are explicitly incompatible with scope/taxonome-ax/v2. All historical live commands used the same physical external ledger and explicit COPILOT_GITHUB_TOKEN environment authentication (useLoggedInUser:false); the private ledger path is represented below as $LEDGER. No direct-provider substitution or repair-loop fallback was used.

• pnpm --silent taxonome eval --cases packages/taxonomy-evaluation/fixtures/inputs.jsonl --gold packages/taxonomy-evaluation/fixtures/gold.json --split all --samples 2 --ledger "$LEDGER" — revised baseline 36/36 cases: 30 eligible SDK predictions, 6 authoritative zero-call preflights, plus 8 deterministic Timeout results.
• pnpm --silent taxonome optimize --cases packages/taxonomy-evaluation/fixtures/inputs.jsonl --gold packages/taxonomy-evaluation/fixtures/gold.json --max-metric-calls 48 --output packages/taxonomy-evaluation/evidence/optimized.v2.ax.json --ledger "$LEDGER" — passed: 49 actual SDK requests, 47 metric entries; real bootstrap plus GEPA. Complete serialization preserves three training-only learned traces, five supervised seed examples, components and statistics.
• The same regression eval command with --artifact packages/taxonomy-evaluation/evidence/optimized.v2.ax.json — 36/36 after normal-path restore/apply.
• pnpm --silent taxonome eval --cases packages/taxonomy-evaluation/fixtures/final-inputs.jsonl --gold packages/taxonomy-evaluation/fixtures/final-gold.json --split heldout --samples 2 --ledger "$LEDGER" — fresh final baseline 12/12 cases; the same command with --artifact packages/taxonomy-evaluation/evidence/optimized.v2.ax.json — 12/12. Each configuration made 10 eligible prompt calls with 2 preflights and 2 additional deterministic Timeout assertions.
• Actual API replay after opaque run/session-ID changes — 3/3, 3 SDK requests.

The final packet was frozen before revised optimization, disjoint by originating run/family, and never supplied to it. The earlier packet is explicitly regression/development evidence. Both comparisons use fixed semantic gold; earlier 33/36 baseline and 35/36 replay failures remain archived. Small synthetic packets do not establish production accuracy.

Historical snapshot: 256/500 requests, 369886 input / 12161 output tokens, no unresolved reservations. It remains byte-for-byte unchanged. The first ID-confounded YAML demo adds 11 calls and is preserved only as historical execution evidence. Current cumulative total: 289/500 after the separate 11-call readable-path control, with 410036 input / 12940 output tokens and 211 slots remaining. Package extraction added zero model calls. Dollar cost remains unknown; SDK cost multipliers and per-call usage are recorded separately.

Documentation and compatibility

Current YAML/archive/plugin implementation:

• Architecture, YAML/ATIF/archive/plugin contracts, API/CLI usage, findings, and compatibility
• Current evidence manifest with frozen historical and fresh YAML-demo provenance
• Runnable custom taxonomy and handler project
• Fresh YAML prompt demo evidence — historical ID-confounded packet
• Opaque-ID prompt control evidence
• Readable-path control evidence
• Package boundary: taxonomy-engine owns runtime mechanisms; ax-taxonomy owns only catalog definitions, fixtures, concrete handlers, and focused tests.

Frozen experiment-head evidence (preserved historical links):

• Original experiment-head architecture/findings
• Original evidence manifest
• Complete selected Ax artifact
• Actual-result Markdown readout: Timeout 2 distinct runs / 4 occurrences; account/query 2 runs / 3 occurrences, not 3 runs.

New private workspace packages only. No Scope API, persistence, Judge, Portal, existing Scope CLI, migration, or production readout integration. Static prompt inventory/manifest/JSONL target registry and dependency notices are updated.

Current private imports use taxonomy-engine for generic APIs and ax-taxonomy for the shipped catalog. createBuiltInHandlerRegistry() and unused hard-coded legacy catalog facades were retired rather than preserved through a cycle or hidden fallback. The wire envelope remains scope/taxonomy-evaluation/v1. Compatibility is explicit rather than silent: run, eval, and optimize now require exact scope/taxonomy/v1 YAML; the normal CLI no longer accepts separate --context; historical v1 evidence remains immutable; the deterministic converter is opt-in; and historical optimization artifacts are rejected under the new contract. Plugins must be explicitly selected compiled JavaScript, run with process permissions, and are not sandboxed. Limits remain explicit: ATIF v1.7, bounded file/HTTP/archive acquisition, SDK-managed sampling/low reasoning, prompt/JSON parsing plus schema validation rather than native constrained decoding, no USD cost estimate or throughput claim.

Checklist

• If Portal features changed, keep CLI capabilities in sync. N/A: no Portal changes.
• If Portal components changed, update their Storybook stories. N/A: no Portal changes.
• If database changes require a migration, include up() / down() and keep it CosmosDB-compatible. N/A: no database changes.
• If dependencies changed, update the lockfile and regenerate NOTICE / NOTICE-REVIEW.txt with pnpm notice as needed. NOTICE-REVIEW.txt unchanged.
• Video showing the behavior before the suggested change — N/A, new CLI.
• Video showing the behavior after the suggested change

[github-actions]

Test Results (Node.js 22)

test: Run #329

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3274	3274	0	0	0	0	0	58.9s

🎉 All tests passed!

Github Test Reporter

[Cedric Vidal (cedricvidal)]

Real Taxonome CLI on the committed synthetic Timeout ATIF fixture: no credentials or Scope services, structured turn/call evidence, and matching repeated-result hashes. This deterministic-path recording makes zero SDK calls; it complements, rather than replaces, the live model evaluation evidence.

terminal.mp4

[github-actions]

Test Results (Node.js 22)

test: Run #330

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3278	3278	0	0	0	0	0	1m39s

🎉 All tests passed!

Github Test Reporter

[github-actions]

Test Results (Node.js 22)

test: Run #331

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3287	3287	0	0	0	0	0	1m7s

🎉 All tests passed!

Github Test Reporter

[github-actions]

Test Results (Node.js 22)

test: Run #332

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3287	3287	0	0	0	0	0	1m34s

🎉 All tests passed!

Github Test Reporter

[Cedric Vidal (cedricvidal)]

Comprehensive Taxonome demo: polished CLI, saved real-SDK evidence, Ax artifact replay, deterministic readout, typed API, and frozen call budget.

terminal.mp4

[github-actions]

Test Results (Node.js 22)

test: Run #357

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3636	3633	3	0	0	0	0	2m11s

❌ Some tests failed!

Name	Failure Message
❌ YAML/archive/plugin CLI process contract > fails concisely without a custom plugin and succeeds with an explicit project-relative plugin	Command failed: /opt/hostedtoolcache/node/22.23.3/x64/bin/node --conditions=source --import tsx /home/runner/work/scope/scope/apps/taxonome/src/index.ts run inputs/run.atif.json --taxonomy taxonomy.yaml --plugin ./dist/handlers/index.js --elements acme-retry-loop taxonome: ENOENT: no such file or directory, realpath '/home/runner/work/scope/scope/examples/taxonome-custom-handler/dist/handlers/index.js'
❌ YAML/archive/plugin CLI process contract > runs the same custom taxonomy/handler against an existing Scope archive	Command failed: /opt/hostedtoolcache/node/22.23.3/x64/bin/node --conditions=source --import tsx /home/runner/work/scope/scope/apps/taxonome/src/index.ts run inputs/run.tar.gz --taxonomy taxonomy.yaml --plugin ./dist/handlers/index.js --elements acme-retry-loop taxonome: ENOENT: no such file or directory, realpath '/home/runner/work/scope/scope/examples/taxonome-custom-handler/dist/handlers/index.js'
❌ YAML/archive/plugin CLI process contract > builds readout JSON/Markdown and verifies the loaded taxonomy digest	Command failed: /opt/hostedtoolcache/node/22.23.3/x64/bin/node --conditions=source --import tsx /home/runner/work/scope/scope/apps/taxonome/src/index.ts run inputs/run.atif.json --taxonomy taxonomy.yaml --plugin ./dist/handlers/index.js --elements acme-retry-loop taxonome: ENOENT: no such file or directory, realpath '/home/runner/work/scope/scope/examples/taxonome-custom-handler/dist/handlers/index.js'

Github Test Reporter

[github-actions]

Test Results (Node.js 22)

test: Run #359

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3636	3636	0	0	0	0	0	1m19s

🎉 All tests passed!

Github Test Reporter

[github-actions]

Test Results (Node.js 22)

test: Run #361

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3640	3640	0	0	0	0	0	1m25s

🎉 All tests passed!

Github Test Reporter

[github-actions]

Test Results (Node.js 22)

test: Run #362

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3641	3641	0	0	0	0	0	1m57s

🎉 All tests passed!

Github Test Reporter

[github-actions]

Test Results (Node.js 22)

test: Run #390

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3626	3626	0	0	0	0	0	2m9s

🎉 All tests passed!

Github Test Reporter