Repository navigation
docs: inventory Loom eval runner responsibilities - #66
Merged
bateau84 merged 1 commit intoOct 7, 2026
Merged
Conversation
bateau84
added a commit
that referenced
this pull request
Oct 7, 2026
## Task Closes #58 — **Task 1 of 9: Define generic orchestration boundary and extraction contract**. This PR is architecture/documentation only. It does not implement the eval engine. ## Inputs integrated Parallel worker results were merged into the integration branch first: - #65 — runner contract/invariant inventory - #66 — Loom eval-runner responsibility inventory Both are based on the post-#45 merged `main` contract. ## Architecture decisions The final synthesis defines: - one `invoke` = exactly one isolated invocation; - retry ownership exclusively in orchestration; - `runtime_evidence/v1` as the only authoritative runtime-evidence source; - explicit separation of product, evidence, and infrastructure outcomes; - generic ownership for planning, iterations, concurrency, attempts, evidence-readiness mechanics, judge lifecycle, tri-state classification, artifacts, and summaries; - project/profile ownership for case semantics, fixtures, prompts, evidence requirements, deterministic assertions, judge meaning, thresholds, and ablation policy; - explicit non-migration of Loom's old direct-container fallback, legacy observer/evidence reconstruction, and superseded runner-safety paths; - concrete internal module decomposition for Tasks 2–5; - concrete internal contracts for normalized cases/jobs, invocation specs, attempt records, retry policy, evidence requirements/readiness, check outcomes, semantic decisions, and durable artifacts; - initial standard/runtime concurrency-lane behavior; - versioned internal run/artifact schemas; - explicit Task 2–5 handoff boundaries. No universal assertion DSL, public `eval` CLI, input JSON API, Loom migration, or skill-ablation implementation is introduced. ## Files - `docs/eval-engine-architecture.md` — final Task 1 synthesis - `docs/eval-engine-runner-contracts.md` — worker B inventory - `docs/loom-eval-runner-responsibility-inventory.md` — worker A inventory ## Branch topology - integration branch: `eval-engine/01-architecture` - base: `main` - worker branches were merged into the integration branch before synthesis. ## Acceptance The architecture is intended to let Tasks #59–#62 proceed independently without redesigning the ownership boundary. Documentation only; no runtime behavior changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of #58 (Task 1/9), parallel worker A.
Adds a focused inventory of the current Loom eval harness from:
The inventory traces and classifies:
It separates each responsibility into generic orchestration, Loom/project semantics, low-level invoke behavior, or legacy/compatibility behavior that should not migrate.
Post-#45 validation
Cross-checked against the merged runner contracts on the integration base:
The document explicitly marks Loom's older direct-container fallback, observer/tool-result reconstruction, runner-safety adapter, and Loom-side evidence_safety eligibility machinery as non-migration paths.
Scope
Documentation only.
No runtime changes, no Loom migration, no final engine API/module design, no universal assertion DSL, and no public eval CLI.
PR target: eval-engine/01-architecture.