refactor: scope eval trust to reviewed runtime observations

[bateau84]

Integrated direction

This PR now integrates the completed child work from #46–#51 into one trusted-checkout runtime-evidence implementation on stock OpenCode 2.0.23.

The normal Loom eval profile remains a trusted-checkout evaluation. This PR does not add an OpenCode patch/fork, remote PluginHost, protected channel, HMAC/signing boundary, capability broker, or hostile-plugin isolation.

Canonical public contract

The only authoritative public runtime-evidence object is:

opencode-eval-runner/runtime-evidence/v1

Raw observer records are internal adapter input only.

Existing tools, actions, tool_result_evidence, stdout/stderr, Session/model text, and workspace files remain convenience/diagnostic data and are not promoted into runtime evidence. Final cleanup #52 also removes competing runtime evidence_eligible signals from the diagnostic tool-result/safety projections.

There is one canonical builder/validator that owns:

• complete | incomplete | unsupported | invalid;
• capture closure and unknown coverage;
• missing terminals;
• observer/callback loss;
• timeout/interruption;
• duplicate/ambiguous invocation and sequence detection;
• identity-based concurrency;
• per-boundary support;
• assertion-scoped boundary/field eligibility.

Unknown coverage is never converted to zero.

Boundary semantics

The v1 contract exposes:

• native
• code_mode_execution
• code_mode_finality

Overall complete means the supported capture boundaries are complete. It does not mean every possible assertion is supported.

This allows:

overall: complete / eligible
native: complete / eligible
code_mode_execution: complete / eligible
code_mode_finality: unsupported / ineligible

An assertion that only requires native evidence can therefore remain eligible even though Code Mode finality is unsupported. Assertions that require a redacted/omitted/unsupported exact field remain ineligible for that field.

Native observation

The integrated observer preserves #51's stock-2.0.23 boundary:

decoded executable input -> transformed tool.execute wrapper

terminal success/error ->
  session.tool.success
  session.tool.failed

It records opaque invocation identity, actual tool, agent, Session, message, real CallID, executable input, Session ancestry, terminal result/error, and monotonic start/terminal ordering.

Correlation is identity-based. It does not use FIFO, input equality, tool name, or completion order.

The raw former native_tool_observations projection is no longer a public result.

Code Mode

The #50 stock-runtime result is retained faithfully.

Supported facts:

• unique per-inner invocation identity;
• actual effective tool;
• decoded/executable input;
• Session/message/agent;
• actual outer execute CallID;
• binding to the outer invocation;
• start/handler-terminal ordering;
• identical concurrent calls and reverse completion.

The transformed handler's earlier value/error is not promoted as caller-final evidence.

Exact final script-visible value/error is represented as:

{
  "state": "unsupported",
  "reason": "stock_codemode_final_boundary_not_exposed"
}

Stock OpenCode 2.0.23 does not expose a supported boundary that proves the exact final value/error seen by a Code Mode script for each inner call. That assertion is reported as unsupported.

Evidence safety

The #47 safety behavior is applied to authoritative dynamic values before their first observation persistence/output sink:

raw value in observer memory
  -> sanitize/redact/omit
  -> size decision
  -> internal capture
  -> validate/account
  -> result serialization
  -> stdout/host-file persistence

Public field states are only:

• available
• redacted
• omitted
• unsupported

Product exit_code, timeout, and product success/failure remain separate from evidence eligibility.

Child PR disposition

PR	Disposition
#46 fail-closed runtime evidence accounting	Incorporated / superseded as a separate implementation. Its stronger accounting is folded into the canonical v1 builder/validator. No second public/accounting status engine remains.
#47 evidence safety	Incorporated. Pre-sink sanitization, redaction/omission, size ordering, credential inventory, and output guard behavior are retained. Internal exact terminology is not part of the public runtime-evidence schema.
#48 provider-free acceptance gate	Incorporated and tightened. It now requires the exact final v1 contract rather than rollout-compatible alternate shapes.
#49 runtime evidence v1 contract	Incorporated and evolved. It remains the public wire contract; overall vs boundary/assertion eligibility was revised to preserve partial support.
#50 stock Code Mode experiment	Incorporated as capability proof/diagnostic probe. Supported execution facts are used; exact final caller value/error remains explicitly unsupported.
#51 stock native observer	Incorporated / superseded as a public projection. Its stock observer boundary is retained as internal adapter input to runtime_evidence; no public native_tool_observations object remains.

Child PRs #46–#51 have no remaining implementation authority after this integration. Final cleanup #52 removes parallel-work duplication without adding architecture. Explicit unsupported areas are: Code Mode caller-final value/error on stock 2.0.23, OpenCode runtime observation for the github-copilot-cli transport, and protection against a deliberately hostile plugin sharing the trusted OpenCode process.

Final integration cleanup

#52 is the final child PR targeting this branch. Its head a3f575d78f4b14d62b76d14dceea49b76496fd2d:

• removes duplicate runtime-evidence constants and dead truncated accounting;
• removes competing evidence_eligible signals from diagnostic tool_result_evidence / safety projections;
• removes the unused legacy tool-result clipping helper;
• confirms PR [SUPERSEDED by #45] feat: preserve eval:live semantics while observing runtime tools #41 protected-runtime/signing/broker machinery is absent;
• documents Code Mode finality, Copilot runtime observation, and hostile-plugin isolation as explicit unsupported areas;
• keeps stock OpenCode 2.0.23 and normal invoke behavior.

Validation on #52:

• CI #297 / run 37521327070: PASS, 93/93 unit tests;
• provider-free runtime evidence acceptance Export tool call arguments in normalized eval results #11 / run 37521327281: PASS, 6/6 helper tests and all 10 scenarios;
• stock native observer integration fix: resolve eval agents with opencode debug agents #23 / run 37521327205: PASS.

#52 has now been squash-merged into this branch as b10c52d50aaf0c038e77e80eaca3890347d5fa82.

Final-head validation

Head: b10c52d50aaf0c038e77e80eaca3890347d5fa82

• CI #296 / run 37519197332 — PASS
  • Python/unit suite: 92/92 PASS
  • stock Code Mode probe: PASS
  • exact caller terminal capability reported unsupported: PASS
  • OpenCode plugin activation preflight: PASS
  • config-root plugin dependency preflight: PASS
  • pinned OpenCode remains 2.0.23
• Provider-free runtime evidence acceptance fix: seed host OpenCode model catalog by default #10 / run 37519197280 — PASS
  • helper/unit checks: 5/5 PASS
  • native_success: PASS
  • native_error: PASS
  • code_success: PASS
  • code_caught_error: PASS
  • concurrent_reverse: PASS
  • delegation: PASS
  • timeout: PASS
  • interrupted: PASS
  • redaction: PASS
  • collector: PASS
• Stock native observer integration fix: invoke OpenCode API with V2 CLI syntax #22 / run 37519197390 — PASS
  • stock 2.0.23: PASS
  • complete capture: PASS
  • provider-free probe: PASS

Review state

The integration cleanup from #52 is merged. PR #45 now contains the final integrated implementation and remains ready for final independent review.

PR #45 remains open/draft and is not merged.

Independent final review

Final review of head b10c52d50aaf0c038e77e80eaca3890347d5fa82 found one blocking correctness defect: stock Code Mode's synthetic model-facing execute invocation was not emitted as an authoritative native observation, so native coverage could falsely remain complete/zero while Code Mode had run and inner parents resolved only to an internal token.

Corrective child PR #54 observes the real outer call at stock execute.before, binds inner parents to that observed invocation, and makes dangling/wrong-identity parents invalid. #54 is green across CI, the stock native probe, and the 10-scenario provider-free acceptance gate.

Review status: NOT READY until #54 is merged. No threat-model expansion is required.

[bateau84]

Supersession checkpoint

• PR [SUPERSEDED by #45] feat: preserve eval:live semantics while observing runtime tools #41: closed as superseded; branch retained for selective implementation/test reuse.
• PR [SUPERSEDED by #45] docs: plan TRUST-001 experiment on stock OpenCode #43: closed as superseded; branch retained for research/provenance.
• Normal profile: trusted-checkout runtime observation, not hostile-plugin resistance.
• Runtime target: stock OpenCode 2.0.23.

CI #265 is green on 618b2e9.

Verified on this replacement head:

• Python/unit checks
• stock OpenCode 2.0.23 image build
• OpenCode plugin activation preflight
• config-root plugin workspace dependency resolution
• pinned tool/version checks
• reasoning-control checks

The old #41 implementation is intentionally not merged wholesale. Evidence-safety, disposable-state, observation semantics, delegation/concurrency probes, and other useful pieces will be ported only where they fit the new stock-runtime trust model. Patched-runtime, protected-channel, remote PluginHost/isolation, HMAC trust-boundary, and signing-as-TRUST-prerequisite work stay out.

[bateau84]

Final independent review — NOT READY

BLOCKING — outer Code Mode execute invocation is missing from authoritative evidence

container/native_observer.ts observes registered direct tools through tool.transform, but stock OpenCode creates the synthetic Code Mode execute tool later inside Tool.snapshot; it is therefore not transform-wrapped.

The observer's execute.before hook only creates/updates an internal activeOuter correlation token. It does not emit a public/native start record for the actual outer execute invocation, and Session terminal handling ignores the outer terminal because there is no corresponding nativeActive entry.

Consequences:

• a Code Mode run can have an actual model-facing execute tool invocation while the public native boundary reports zero such calls and remains complete;
• an absence assertion over native/direct tools can therefore be false-positive for execute;
• Code Mode parent points to an internal opaque token that has no corresponding outer observation in runtime_evidence;
• the acceptance gate currently only checks that this parent ID is non-empty/shared; it does not require a matching observed outer invocation.

This conflicts with the v1 contract's description of native as direct/native execution and its claim of parent binding to the actual outer invocation.

Required correction

Observe the model-facing outer execute call through a supported stock-2.0.23 runtime surface and include it in authoritative evidence, or explicitly add a separate unsupported/partial outer-execute boundary so absence cannot be claimed.

Do not infer it from product stdout/model data and do not patch OpenCode. Stock Session tool input/called/terminal surfaces or another reviewed supported boundary should be investigated.

Add provider-free regression coverage that proves:

1. one Code Mode run contains an authoritative outer execute observation;
2. inner records bind to that exact observed outer invocation;
3. a no-native/absence assertion cannot report zero when execute actually ran;
4. concurrent inner calls retain the same observed outer parent.

Other review results

The scoped threat model is otherwise preserved: I found no reintroduction of protected-channel, signing, broker, OpenCode patching, or hostile-plugin isolation. The native Session terminal boundary, fail-closed accounting, explicit Code Mode finality limitation, pre-sink evidence projection, host re-validation, and the final-head CI/acceptance runs are coherent.

Verdict: NOT READY until the outer-execute coverage hole is closed or explicitly represented as unsupported.

[bateau84]

Final re-review — NOT READY

#54 is merged and closes the outer Code Mode execute coverage defect. The integrated head is green across CI, provider-free acceptance, and the stock native observer gate.

BLOCKING — authoritative capture is still target-writable

The observer still writes authoritative records to:

/tmp/runtime/runtime-observer.jsonl

runner/cli.py mounts all of /tmp as a writable tmpfs for the evaluated OpenCode process. Normal model-driven tool/shell subprocesses execute under that same runtime user and can therefore potentially read, truncate, replace, or append to this file.

This conflicts with the trusted-checkout contract already documented in this PR: target-writable evidence files do not independently establish that an event occurred.

This is not hostile-plugin resistance and does not require the old TRUST-001 security architecture. It is ordinary eval correctness: model/tool execution must not be able to manufacture or alter the evidence used to score itself.

Required correction

Move authoritative observer transport off the evaluated process's writable filesystem. A small runner-owned pipe/socketpair/inherited-FD style transport is sufficient if normal tool subprocesses cannot write to it. No signing, HMAC, broker, remote PluginHost, or OpenCode patch is required.

Add one provider-free tamper regression proving an evaluated tool/subprocess cannot create, replace, truncate, or append authoritative records.

Status

• outer execute authoritative observation: FIXED
• inner-to-outer binding: FIXED
• Code Mode finality remains correctly unsupported
• stock OpenCode 2.0.23: PASS
• CI / acceptance / native observer gates: PASS
• authoritative transport integrity against normal evaluated tool writes: BLOCKING

Verdict: NOT READY until the target-writable capture file is removed from the authoritative path.

[bateau84]

Final blocker follow-up is now green in #55.

• fix: observe outer Code Mode execute evidence #54 is merged into refactor/trusted-checkout-evidence and closes the authoritative outer Code Mode execute blocker.
• fix: close final runtime evidence blockers #55 is rebased directly on that base and removes the target-writable authoritative capture file in favor of a runner-owned one-connection loopback stream.
• The evaluated-shell tamper regression creates/deletes/recreates/appends the old /tmp/runtime/runtime-observer.jsonl path and passes only when the forged record cannot enter authoritative runtime_evidence.

Current #55 validation:

• CI #329: PASS, 95/95 tests, Code Mode probe PASS, stock OpenCode 2.0.23.
• Provider-free acceptance [SUPERSEDED by #45] docs: plan TRUST-001 experiment on stock OpenCode #43: PASS, all 11/11 scenarios including capture_tamper.
• Stock native observer integration fix: close final runtime evidence blockers #55: PASS.

Both final review blockers are closed by the combined base + #55 state. PR #45 remains open/unmerged pending integration/review of #55.

[bateau84]

Final independent review — READY TO MERGE

Reviewed integrated head:

e3eb017ff070f3956119472a7d0fb74faa247102

Both previously blocking correctness defects are closed:

1. Outer Code Mode execute authority — the real model-facing synthetic execute invocation is observed at stock execute.before, settled by Session terminal events, and inner Code Mode calls bind to that exact observed outer invocation.
2. Target-writable capture authority — production no longer consumes /tmp/runtime/runtime-observer.jsonl; authoritative records cross a runner-owned one-connection loopback stream, the listener closes after the trusted observer connects, and evaluated subprocess writes to the old path cannot affect evidence.

Final-head validation:

• CI #330: PASS
• Provider-free runtime evidence acceptance #44: PASS, including capture_tamper
• Stock native observer integration #56: PASS
• stock OpenCode 2.0.23 retained

The canonical runtime_evidence/v1 contract remains the sole eligibility authority. Missing/lost/malformed capture fails closed. Product outcome remains separate from evidence outcome. Exact Code Mode caller-final value/error remains explicitly unsupported. No OpenCode patch, signing/HMAC, protected channel, broker, remote PluginHost, or hostile-plugin isolation architecture has been reintroduced.

No BLOCKING, IMPORTANT, or MINOR correctness findings requiring changes remain.

Verdict: READY TO MERGE #45.

[bateau84]

Focused Product Readiness re-pass — PRODUCT READY

Reviewed integrated head:

a67c931696d1ae2b2d16f49beab6c01c912aa02b

PR #56 is merged and closes the only Product Readiness blocker identified in the previous gate.

Evidence-readiness decision procedure

The public contract now gives consumers an explicit assertion-level procedure:

1. validate runtime_evidence/v1;
2. overall incomplete or invalid makes every runtime assertion ineligible for PASS;
3. declare the boundary/boundaries required by the assertion and require each to be complete;
4. if an assertion needs an exact field, require that field to be available; redacted/omitted are incomplete for that assertion and unsupported is unsupported;
5. never backfill authoritative facts from tools, actions, tool_result_evidence, stdout/stderr, model text, or workspace files.

This is sufficient for Loom to make evidence-readiness decisions without moving judging semantics into the runner.

Product Readiness findings

BLOCKING: none.

IMPORTANT non-blocking follow-ups:

• Loom must migrate its verdict path to consume runtime_evidence before using this capability to authorize PASS.
• Aggregate capture-limit failures currently diagnose generically; dedicated capture_size_limit / capture_record_limit reason codes would improve operations but cannot create false PASS.
• Any OpenCode upgrade beyond reviewed stock 2.0.23 should re-run the observer capability probes as an upgrade gate.

Accepted release limitations

• exact Code Mode script-visible final value/error remains unsupported;
• github-copilot-cli runtime observation remains unsupported;
• deliberately malicious same-process evaluated plugins remain outside the trusted-checkout profile;
• stock OpenCode 2.0.23 is the reviewed runtime.

Compatibility / migration

The README now states that runtime_evidence is mandatory, host and image form a compatibility pair, and legacy/custom images without valid v1 evidence are rejected. The primary Result contract therefore matches actual host behavior.

Exact-head validation

• CI #332: PASS — 95/95 tests
• Provider-free acceptance #45: PASS — all 11 scenarios including capture_tamper
• Stock native observer #57: PASS
• stock OpenCode 2.0.23 confirmed

No additional Product Readiness pass is required for PR #45 unless the runtime or public contract changes again. Loom should perform its own integration/acceptance pass when its scoring path is migrated.

PRODUCT READY — PR #45 may merge