feat: observation criteria with taxonomy and run/iteration subject (#1156)

[Cedric Vidal (cedricvidal)]

Original author: Cedric Vidal (@cedricvidal)

Why

Issue #1156 asked for per-iteration trajectory observations: a way to record what a coding agent did during a run without steering it. This PR generalizes that into observation criteria, reusing the existing criteria entity (with optional dependency support) instead of introducing a new entity.

Stacked on #1147 (DAG-based post-processing pipeline, base main). Until #1147 merges, this PR's diff includes that foundation. The work unique to this branch starts at commit fc262735 (feat(observations): ...) and the 7 #1156 subject commits on top.

What changed

1. Observation criteria (kind: "gate" | "observation", default gate)
Gates steer the agent's session via pass/fail feedback in the gate DAG. Observations are record-only: they produce a boolean + evidence per run and never feed back to the agent. Observations are excluded from validateGateConfigs, gate-satisfaction, and feedback selection. Migration 021 backfills existing criteria to gate (mirrors the gates backfill).

2. Static taxonomy classification (taxonomyElementId?)
Optional design-time classification against a hard-coded enum of namespaced element IDs (e.g. dimension:idiomatic-use), validated on write. No first-class taxonomy entity yet — namespaced string IDs keep a future migration clean.

3. Criterion subject (subject: "run" | "iteration")
Controls evaluation granularity and where results are stored:

• run — evaluated once over the whole merged run trajectory, stored in run.observationResults
• iteration — evaluated per turn in isolation, stored in turn.observationResults

The criteria AI assistant auto-detects the subject; the user can override it in the portal/CLI; the judge honors it. Migration 022 backfills subject.

4. Whole-run trajectory in the judge
The judge merges the agent trajectory from atifUrls[] (all turns) so run-subject observations can see cross-iteration behavior a per-turn evaluation cannot (e.g. "scaffolded with Vite in iteration 1, ran tests in iteration 2"). Trajectory navigation was unified into one source-agnostic tool set, with ATIF tool listing/detail lookups and large-output spill-to-file for huge tool outputs.

5. pp-taxonomy post-processing handler
A new post-processor handler evaluates observation criteria via the judge as part of the handler DAG, splitting iteration- vs run-subject criteria and writing each result to the right place.

6. Portal + CLI
Portal: observation kind/taxonomy UI, a subject toggle, a submit-run observation picker, a Taxonomy tab (with an Observations subsection) on run detail, an Enrichment badge aggregated over the handler DAG, and copy clarifying that gates steer the agent's discussion while observations do not. CLI reaches parity with a --subject flag and whole-run observation support.

Testing

Unit + integration tests added across shared, api, judge, scheduler, post-processor, and portal, plus a live ATIF-trajectory observation acceptance test. End-to-end validation against the shared CosmosDB dev stack (import a real integration run, seed run- and iteration-subject observations, assert the run-subject criterion sees cross-iteration behavior and does not leak into per-turn results) is in progress.

Non-goals

No first-class taxonomy entity (deferred), single element per criterion (no multi-lens list), no taxonomy versioning, booleans only (no numeric scores — fractions emerge by aggregation across runs), and no per-run reclassification.

Fixes: #1156

[github-actions]

Test Results (Node.js 22)

test: Run #241

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3365	3359	6	0	0	0	0	1m32s

❌ Some tests failed!

Name	Failure Message
❌ API Endpoints > POST /api/v1/criteria > creates an observation criterion with a taxonomy element	expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > defaults an observation's subject to 'run'	expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > creates an observation with an explicit subject:'iteration' override	expected 400 to be 201 // Object.is equality
❌ PostProcessor > re-processes stale versions and stamps success for the same run attempt	Cannot read properties of undefined (reading 'collection')
❌ PostProcessor > claims a run left in the scheduler's 'queued' state and drives it to done	Cannot read properties of undefined (reading 'collection')

See all failed tests here

Github Test Reporter

[github-actions]

Test Results (Node.js 22)

test: Run #282

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
3367	3361	6	0	0	0	0	1m11s

❌ Some tests failed!

Name	Failure Message
❌ API Endpoints > POST /api/v1/criteria > creates an observation criterion with a taxonomy element	expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > defaults an observation's subject to 'run'	expected 400 to be 201 // Object.is equality
❌ API Endpoints > POST /api/v1/criteria > creates an observation with an explicit subject:'iteration' override	expected 400 to be 201 // Object.is equality
❌ PostProcessor > re-processes stale versions and stamps success for the same run attempt	Cannot read properties of undefined (reading 'collection')
❌ PostProcessor > claims a run left in the scheduler's 'queued' state and drives it to done	Cannot read properties of undefined (reading 'collection')

See all failed tests here

Github Test Reporter

[Wassim Chegham (manekinekko)]

Read-only review of a9e9dbd. Findings are inline; the symlink security observation is explicitly pre-existing, not a regression introduced by this PR.

[Wassim Chegham (manekinekko)]

[P1] Keep the CriteriaBadge import

The ID column still renders <CriteriaBadge> at line 168, but this replacement removes its import. The Portal build fails with TS2552: Cannot find name 'CriteriaBadge'. Please retain CriteriaBadge alongside the two new badge imports.

[Wassim Chegham (manekinekko)]

[P1] Scope observation reads and writes to the claimed run attempt

A request can be retried while this judge call is in flight because the coding run is already terminal. This update filters only by request ID, so the old attempt's result is then written into the replacement run. I reproduced this with the production handler by swapping run._id during judge.evaluate: the new pending attempt received the old attempt's observations. The per-turn write has the same issue when iteration numbers overlap.

The worker's initial claim and guarded terminal-status write do not protect these intermediate result writes. Please include message.runId in the initial read and every result-update filter, and handle a zero-match update without modifying the new attempt.

[Wassim Chegham (manekinekko)]

[P1] Supply projectId in the new observation endpoint tests

These new POSTs omit the required project query parameter. Consequently, the three happy-path tests return 400 instead of 201, and the new rejection tests return 400 for missing scope before they exercise taxonomy/subject validation. Running the endpoint suite produces three failures from these additions.

Please add ?projectId=${TEST_PROJECT_ID} to these requests and use the existing request(testServer()) helper so the tests reach the intended behavior without creating extra servers.

[Wassim Chegham (manekinekko)]

[P2] Wire the observation fields through the actual API seed path

This helper now preserves observation metadata, but POST /api/v1/criteria/seed does not call it: apps/api/src/routes/criteria.ts still performs an inline upsert whose $setOnInsert includes only the old fields. The shared input schema accepts kind, subject, and taxonomyElementId, then that route discards all three. I reproduced this by invoking the registered seed handler with a valid observation payload.

As a result, the updated CLI import/YAML mapper still creates legacy gate criteria, not observations. Please update the actual route to preserve and validate these fields (or route it through the store) and cover an import/seed round trip.

[Wassim Chegham (manekinekko)]

[P2] Preserve observations in profile fan-out and resubmission paths

The profile-variation branch returns earlier, before this validation and the new persistence logic. Its request-document construction never includes observations, so a comparative submission with a valid observation selection returns 201 while every created run silently loses that selection. I reproduced this with the registered handler: both the base and variation requests omitted the field.

Please resolve/validate the shared selection before branching and include it on every fan-out document. The bulk-resubmit constructor later in this file also needs to preserve original.observations; it currently drops the selection as well.

[Wassim Chegham (manekinekko)]

[P2] Preserve the new observation metadata in Portal YAML exports

The existing Export actions in CriteriaList and CriterionDetail still call criterionToYaml in apps/portal/src/lib/criteria-export.ts, which emits only id, prompt, and depends_on. Exporting a criterion with these new fields therefore drops its observation kind, subject, and taxonomy before it even reaches an importer; a subsequent import defaults it to a gate. I confirmed this with the production exporter.

Please serialize the new metadata there, matching the updated CLI exporter, and add an observation round-trip test.

[Wassim Chegham (manekinekko)]

[P2] Add an explicit way to clear an existing taxonomy classification

Choosing Unclassified sets this value to undefined, which JSON.stringify removes from the update body. The store interprets the missing field as no change and retains the previous taxonomy. Sending null is not an alternative because the update schema rejects it. I reproduced both behaviors with the store and schema.

Please define an explicit clear/unset value through the UI, API schema, and store, and translate it into $unset. Otherwise the new Unclassified option cannot persist, and a classified observation cannot be cleanly converted back to a gate either.

[Wassim Chegham (manekinekko)]

[P2] Treat taxonomy blocked by an ATIF failure as terminal

With observations configured and handlerStatus = { 'pp-atif': { status: 'failed' } }, taxonomyPending is true and this function returns processing before considering the failure. The scheduler deliberately never dispatches taxonomy after its dependency fails, so that pending state cannot resolve. The badge remains Enriching and RunDetail keeps polling indefinitely.

Please account for blocked descendants, as the scheduler's DAG-drain calculation does, rather than treating every absent taxonomy status as work still in flight.

[Wassim Chegham (manekinekko)]

[P2] Update the static prompt contract for suggestedSubject

The production parser now returns this field, but the committed authoring adapter contract still expects only prompt and suggestedId. The offline adapter suite fails at evaluations/static-prompts/adapter/adapter.test.ts:126 with the additional suggestedSubject: 'run' property.

Please update the corresponding contract and curated subject-classification cases alongside this prompt/parser change. The failing check is reproducible with pnpm --filter static-prompt-evals exec vitest run --config adapter/vitest.config.ts --configLoader runner.

[Wassim Chegham (manekinekko)]

Non-blocking security follow-up: pre-existing symlink escape (HIGH)

This containment check compares normalized path strings, but the later filesystem reads follow symlinks. A link located inside the workspace can therefore pass the check while its resolved target is outside every approved read root. The snapshot creation/extraction path preserves such links, so an agent-created evidence file can potentially expose judge-container files through evaluation feedback.

This is not introduced by this PR: the same bypass and archive behavior exist at merge-base 2ca370ba. The boundary bypass was reproduced using a harmless dependency symlink, and preservation was checked with an in-memory archive; no credentials were read, and production-image extraction or LLM-mediated disclosure was not exercised. Please track a separate hardening fix that resolves roots and targets with realpath (or rejects escaping symlinks), including directory-symlink cases.