Skip to content

fix: observe outer Code Mode execute evidence - #54

Merged
bateau84 merged 11 commits into
refactor/trusted-checkout-evidencefrom
fix/runtime-evidence-outer-execute-final
Oct 6, 2026
Merged

fix: observe outer Code Mode execute evidence#54
bateau84 merged 11 commits into
refactor/trusted-checkout-evidencefrom
fix/runtime-evidence-outer-execute-final

Conversation

@bateau84

@bateau84 bateau84 commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Purpose

Correct the blocking outer-execute evidence gap found during the independent final review of #45.

This branch is based on the post-#52 target head b10c52d50aaf0c038e77e80eaca3890347d5fa82.

Defect

Stock OpenCode 2.0.23 creates the model-facing Code Mode execute tool inside Tool.snapshot, after registration transforms have run. The integrated observer therefore did not emit a native observation for that real outer invocation. Inner records were linked only to an internal opaque token.

That allowed the public native boundary to report complete/zero outer-execute observations while Code Mode had actually run.

Correction

  • Observe the real model-facing outer execute invocation at stock execute.before.
  • Keep session.tool.success / session.tool.failed as terminal authority.
  • Bind Code Mode inner records to that observed outer invocation ID.
  • Reject dangling, wrong-tool, or identity-mismatched parents in canonical accounting/validation.
  • Strengthen provider-free acceptance checks for outer observation, exact parent resolution, identity agreement, non-zero native coverage, concurrency, and reverse completion.
  • Document that synthetic outer execute input is exact at execute.before but pre-CodeMode.Input decode.

Scope

No threat-model expansion. No OpenCode patch/fork, protected channel, signing, broker, process isolation, or hostile-plugin machinery.

Targets refactor/trusted-checkout-evidence.

Validation

Head: 221af580640360c16e263cc05136d3be0d090a0f

#53 is closed as superseded; it became dirty only because #52 merged while this review was in progress.

This was referenced Oct 6, 2026
Closed
Merged
@bateau84
bateau84 merged commit b10c8e9 into refactor/trusted-checkout-evidence Oct 6, 2026
3 checks passed
@bateau84
bateau84 deleted the fix/runtime-evidence-outer-execute-final branch October 6, 2026 21:03
@bateau84 bateau84 mentioned this pull request Oct 6, 2026
Merged
bateau84 added a commit that referenced this pull request Oct 6, 2026
@bateau84
fix: close final runtime evidence blockers (#55)
e3eb017 
## Scope

Focused corrective child of #45, targeting
`refactor/trusted-checkout-evidence`.

The base now includes merged #54 (`b10c8e9`), which closes the
authoritative outer Code Mode `execute` blocker. This PR is rebased
directly on that commit and contains only the remaining
authoritative-capture transport correction plus its regression coverage.

Stock OpenCode remains **2.0.23**. No threat-model expansion.

## Blocker 1 — outer Code Mode `execute`

**Closed in the base by #54 and revalidated here.**

The current integrated runtime evidence:

- observes the real synthetic outer `execute` at the supported stock
`execute.before` boundary;
- records Session, agent, message, real CallID, supported hook input,
and a unique runtime invocation ID;
- uses stock `session.tool.success` / `session.tool.failed` as terminal
authority;
- binds every inner Code Mode call to that exact observed outer
invocation;
- rejects dangling or identity-mismatched parents;
- keeps exact Code Mode caller-finality explicitly unsupported.

The Code Mode probe remains green after the capture-transport change.

## Blocker 2 — remove target-writable authoritative capture

The production observer no longer uses:

`/tmp/runtime/runtime-observer.jsonl`

Authoritative observer records now cross the process boundary through a
**runner-owned one-connection loopback stream**:

1. the runner binds an ephemeral listener on `127.0.0.1` immediately
before the real OpenCode invocation;
2. the trusted observer connects during plugin startup;
3. the listener closes after accepting that one connection;
4. the observer removes the endpoint from `process.env` before evaluated
tool subprocesses run;
5. sanitized observer records flow over the established stream;
6. the runner drains and validates those bytes through the existing
canonical builder/validator/accounting path.

A direct inherited memfd/FD was tried first, but stock
`@opencode/cli@2.0.23` crosses an internal process boundary that does
not preserve arbitrary extra descriptors. The one-connection stream
keeps stock OpenCode unchanged and avoids filesystem authority without
introducing security-platform machinery.

Protection against a deliberately malicious same-process plugin remains
explicitly out of scope.

## Tamper regression

Provider-free acceptance includes an evaluated tool that spawns
`/bin/sh` and:

- creates the old capture path;
- deletes it;
- recreates it;
- appends a forged observer record.

The scenario passes only if the shell tamper completes, authoritative
evidence remains complete/eligible, the real tamper tool is observed,
and the forged invocation/tool never appears in `runtime_evidence`.

## Preserved

- `opencode-eval-runner/runtime-evidence/v1`;
- canonical builder / validator / accounting;
- pre-sink credential sanitization;
- stock Session terminal authority;
- Code Mode exact caller-finality remains unsupported;
- normal `invoke` behavior;
- assertion-scoped eligibility;
- stock OpenCode 2.0.23 only.

No signing/HMAC, protected channel, remote PluginHost, capability
broker, hostile-plugin isolation, or OpenCode patch is introduced.

## Validation

Current clean head is based directly on merged #54 and is mergeable.

- **CI #329 / run 37532684965 — PASS**
  - **95/95 Python tests**
  - stock Code Mode probe: **PASS** (`diagnostics_passed: true`)
  - OpenCode **2.0.23**
- **Provider-free runtime evidence acceptance #43 / run 37532684985 —
PASS**
  - all **11/11** scenarios:
    - native_success
    - native_error
    - code_success
    - code_caught_error
    - concurrent_reverse
    - delegation
    - timeout
    - interrupted
    - redaction
    - collector
    - capture_tamper
- **Stock native observer integration #55 / run 37532684878 — PASS**
  - stock OpenCode 2.0.23
  - capture complete
  - provider-free native observation

## Merge scope

This PR is ready for review against
`refactor/trusted-checkout-evidence`.

Do **not** merge PR #45 as part of this change.
bateau84 added a commit that referenced this pull request Oct 7, 2026
@bateau84
refactor: scope eval trust to reviewed runtime observations (#45)
fd9da10 
## Integrated direction

This PR now integrates the completed child work from #46–#51 into one
trusted-checkout runtime-evidence implementation on stock OpenCode
**2.0.23**.

The normal Loom eval profile remains a trusted-checkout evaluation. This
PR does **not** add an OpenCode patch/fork, remote PluginHost, protected
channel, HMAC/signing boundary, capability broker, or hostile-plugin
isolation.

## Canonical public contract

The only authoritative public runtime-evidence object is:

`opencode-eval-runner/runtime-evidence/v1`

Raw observer records are internal adapter input only.

Existing `tools`, `actions`, `tool_result_evidence`, stdout/stderr,
Session/model text, and workspace files remain convenience/diagnostic
data and are not promoted into runtime evidence. Final cleanup #52 also
removes competing runtime `evidence_eligible` signals from the
diagnostic tool-result/safety projections.

There is one canonical builder/validator that owns:

- `complete | incomplete | unsupported | invalid`;
- capture closure and unknown coverage;
- missing terminals;
- observer/callback loss;
- timeout/interruption;
- duplicate/ambiguous invocation and sequence detection;
- identity-based concurrency;
- per-boundary support;
- assertion-scoped boundary/field eligibility.

Unknown coverage is never converted to zero.

## Boundary semantics

The v1 contract exposes:

- `native`
- `code_mode_execution`
- `code_mode_finality`

Overall `complete` means the supported capture boundaries are complete.
It does **not** mean every possible assertion is supported.

This allows:

```text
overall: complete / eligible
native: complete / eligible
code_mode_execution: complete / eligible
code_mode_finality: unsupported / ineligible
```

An assertion that only requires `native` evidence can therefore remain
eligible even though Code Mode finality is unsupported. Assertions that
require a redacted/omitted/unsupported exact field remain ineligible for
that field.

## Native observation

The integrated observer preserves #51's stock-2.0.23 boundary:

```text
decoded executable input -> transformed tool.execute wrapper

terminal success/error ->
  session.tool.success
  session.tool.failed
```

It records opaque invocation identity, actual tool, agent, Session,
message, real CallID, executable input, Session ancestry, terminal
result/error, and monotonic start/terminal ordering.

Correlation is identity-based. It does not use FIFO, input equality,
tool name, or completion order.

The raw former `native_tool_observations` projection is no longer a
public result.

## Code Mode

The #50 stock-runtime result is retained faithfully.

Supported facts:

- unique per-inner invocation identity;
- actual effective tool;
- decoded/executable input;
- Session/message/agent;
- actual outer `execute` CallID;
- binding to the outer invocation;
- start/handler-terminal ordering;
- identical concurrent calls and reverse completion.

The transformed handler's earlier value/error is **not** promoted as
caller-final evidence.

Exact final script-visible value/error is represented as:

```json
{
  "state": "unsupported",
  "reason": "stock_codemode_final_boundary_not_exposed"
}
```

> Stock OpenCode 2.0.23 does not expose a supported boundary that proves
the exact final value/error seen by a Code Mode script for each inner
call. That assertion is reported as unsupported.

## Evidence safety

The #47 safety behavior is applied to authoritative dynamic values
before their first observation persistence/output sink:

```text
raw value in observer memory
  -> sanitize/redact/omit
  -> size decision
  -> internal capture
  -> validate/account
  -> result serialization
  -> stdout/host-file persistence
```

Public field states are only:

- `available`
- `redacted`
- `omitted`
- `unsupported`

Product `exit_code`, timeout, and product success/failure remain
separate from evidence eligibility.

## Child PR disposition

| PR | Disposition |
| --- | --- |
| #46 fail-closed runtime evidence accounting | **Incorporated /
superseded as a separate implementation.** Its stronger accounting is
folded into the canonical v1 builder/validator. No second
public/accounting status engine remains. |
| #47 evidence safety | **Incorporated.** Pre-sink sanitization,
redaction/omission, size ordering, credential inventory, and output
guard behavior are retained. Internal `exact` terminology is not part of
the public runtime-evidence schema. |
| #48 provider-free acceptance gate | **Incorporated and tightened.** It
now requires the exact final v1 contract rather than rollout-compatible
alternate shapes. |
| #49 runtime evidence v1 contract | **Incorporated and evolved.** It
remains the public wire contract; overall vs boundary/assertion
eligibility was revised to preserve partial support. |
| #50 stock Code Mode experiment | **Incorporated as capability
proof/diagnostic probe.** Supported execution facts are used; exact
final caller value/error remains explicitly unsupported. |
| #51 stock native observer | **Incorporated / superseded as a public
projection.** Its stock observer boundary is retained as internal
adapter input to `runtime_evidence`; no public
`native_tool_observations` object remains. |

Child PRs #46–#51 have no remaining implementation authority after this
integration. Final cleanup #52 removes parallel-work duplication without
adding architecture. Explicit unsupported areas are: Code Mode
caller-final value/error on stock 2.0.23, OpenCode runtime observation
for the `github-copilot-cli` transport, and protection against a
deliberately hostile plugin sharing the trusted OpenCode process.

## Final integration cleanup

#52 is the final child PR targeting this branch. Its head
`a3f575d78f4b14d62b76d14dceea49b76496fd2d`:

- removes duplicate runtime-evidence constants and dead `truncated`
accounting;
- removes competing `evidence_eligible` signals from diagnostic
`tool_result_evidence` / safety projections;
- removes the unused legacy tool-result clipping helper;
- confirms PR #41 protected-runtime/signing/broker machinery is absent;
- documents Code Mode finality, Copilot runtime observation, and
hostile-plugin isolation as explicit unsupported areas;
- keeps stock OpenCode 2.0.23 and normal `invoke` behavior.

Validation on #52:
- CI #297 / run 37521327070: **PASS**, 93/93 unit tests;
- provider-free runtime evidence acceptance #11 / run 37521327281:
**PASS**, 6/6 helper tests and all 10 scenarios;
- stock native observer integration #23 / run 37521327205: **PASS**.

#52 has now been squash-merged into this branch as
`b10c52d50aaf0c038e77e80eaca3890347d5fa82`.

## Final-head validation

Head: `b10c52d50aaf0c038e77e80eaca3890347d5fa82`

- **CI #296 / run 37519197332 — PASS**
  - Python/unit suite: **92/92 PASS**
  - stock Code Mode probe: **PASS**
  - exact caller terminal capability reported unsupported: **PASS**
  - OpenCode plugin activation preflight: **PASS**
  - config-root plugin dependency preflight: **PASS**
  - pinned OpenCode remains **2.0.23**
- **Provider-free runtime evidence acceptance #10 / run 37519197280 —
PASS**
  - helper/unit checks: **5/5 PASS**
  - `native_success`: PASS
  - `native_error`: PASS
  - `code_success`: PASS
  - `code_caught_error`: PASS
  - `concurrent_reverse`: PASS
  - `delegation`: PASS
  - `timeout`: PASS
  - `interrupted`: PASS
  - `redaction`: PASS
  - `collector`: PASS
- **Stock native observer integration #22 / run 37519197390 — PASS**
  - stock 2.0.23: PASS
  - complete capture: PASS
  - provider-free probe: PASS

## Review state

The integration cleanup from #52 is merged. PR #45 now contains the
final integrated implementation and remains ready for final independent
review.

PR #45 remains open/draft and is **not merged**.


## Independent final review

Final review of head `b10c52d50aaf0c038e77e80eaca3890347d5fa82` found
one blocking correctness defect: stock Code Mode's synthetic
model-facing `execute` invocation was not emitted as an authoritative
native observation, so native coverage could falsely remain
complete/zero while Code Mode had run and inner parents resolved only to
an internal token.

Corrective child PR #54 observes the real outer call at stock
`execute.before`, binds inner parents to that observed invocation, and
makes dangling/wrong-identity parents invalid. #54 is green across CI,
the stock native probe, and the 10-scenario provider-free acceptance
gate.

**Review status: NOT READY until #54 is merged.** No threat-model
expansion is required.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bateau84