Repository navigation
docs: inventory runner contracts for eval-engine extraction - #65
Merged
bateau84 merged 1 commit intoOct 7, 2026
Merged
Conversation
bateau84
force-pushed
the
eval-engine/01-runner-contracts
branch
from
October 7, 2026 08:16
45946d8 to
29b2b4d
Compare
bateau84
added a commit
that referenced
this pull request
Oct 7, 2026
## Task Closes #58 — **Task 1 of 9: Define generic orchestration boundary and extraction contract**. This PR is architecture/documentation only. It does not implement the eval engine. ## Inputs integrated Parallel worker results were merged into the integration branch first: - #65 — runner contract/invariant inventory - #66 — Loom eval-runner responsibility inventory Both are based on the post-#45 merged `main` contract. ## Architecture decisions The final synthesis defines: - one `invoke` = exactly one isolated invocation; - retry ownership exclusively in orchestration; - `runtime_evidence/v1` as the only authoritative runtime-evidence source; - explicit separation of product, evidence, and infrastructure outcomes; - generic ownership for planning, iterations, concurrency, attempts, evidence-readiness mechanics, judge lifecycle, tri-state classification, artifacts, and summaries; - project/profile ownership for case semantics, fixtures, prompts, evidence requirements, deterministic assertions, judge meaning, thresholds, and ablation policy; - explicit non-migration of Loom's old direct-container fallback, legacy observer/evidence reconstruction, and superseded runner-safety paths; - concrete internal module decomposition for Tasks 2–5; - concrete internal contracts for normalized cases/jobs, invocation specs, attempt records, retry policy, evidence requirements/readiness, check outcomes, semantic decisions, and durable artifacts; - initial standard/runtime concurrency-lane behavior; - versioned internal run/artifact schemas; - explicit Task 2–5 handoff boundaries. No universal assertion DSL, public `eval` CLI, input JSON API, Loom migration, or skill-ablation implementation is introduced. ## Files - `docs/eval-engine-architecture.md` — final Task 1 synthesis - `docs/eval-engine-runner-contracts.md` — worker B inventory - `docs/loom-eval-runner-responsibility-inventory.md` — worker A inventory ## Branch topology - integration branch: `eval-engine/01-architecture` - base: `main` - worker branches were merged into the integration branch before synthesis. ## Acceptance The architecture is intended to let Tasks #59–#62 proceed independently without redesigning the ownership boundary. Documentation only; no runtime behavior changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Worker B for #58 (Task 1/9): post-PR-#45 runner contract/invariant inventory.
This PR is intentionally documentation-only. It does not implement the eval engine, change
invoke, migrate Loom, add a publicevalCLI, or define the final Task 1 architecture.What it captures
invoke= exactly one isolated invocation;opencode-eval-runner/v1result boundary;runtime_evidence/v1;invoke, cross-checked against Loom's current narrow retry policy;Validation
Cross-checked against the landed post-#45 contract:
main/ PR refactor: scope eval trust to reviewed runtime observations #45 merge commitfd9da10cbe2a8182fc8910ec199a221250deb3ca25478106773969870931af3fb34e7b2586746606runner/cli.pycontainer/invoke.pydocs/invocation-usage.mddocs/runtime-evidence-contract.mdaction.ymlfunctionality-anchor-requirements-coherancescripts/run-evals.pyblob30f2ae87764be180173a9e5b4b609ee79b2f063deval-engine/01-architecturenow matchesmainat the PR #45 merge commit. The worker branch has been rebased onto that exact state, leaving a one-file documentation diff.Part of #58.