Skip to content

Add frozen benchmark comparisons and evidence-backed refinement - #4

Open
Ker102 wants to merge 2 commits into
mainfrom
feature/benchmark-evidence-and-refinement
Open

Ker102 wants to merge 2 commits into
mainfrom
feature/benchmark-evidence-and-refinement

Conversation

@Ker102

@Ker102 Ker102 commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Harneloop can now compare a frozen candidate against its baseline without silently mixing models, budgets, task sets or edited evidence. New benchmark commands pin source/grader revisions and candidate contents, require distinct artifact-backed task/trial runs, and report paired wins, losses, ties and unavailable evaluations.

Compatible candidates can be composed with fresh validation, and operators can retain lessons linked to finished-run artifacts. New run/evidence fingerprints are checked again before evidence-backed promotion, so changing the overlay or captured result requires fresh evidence. Legacy records remain readable.

Includes a CLI workflow guide and schema additions. Executors, model-provider clients, benchmark data, videos and raw outputs remain outside the installed framework. These checks verify local integrity and operator-attested conditions; they do not authenticate remote model execution or enforce a solver sandbox.

Validation:

  • python -m unittest discover -s tests: 102 tests, successful, one POSIX executable-mode check skipped on Windows.
  • python -m compileall -q src tests and CLI smoke checks passed.
  • A separate frozen 16-task HumanEval+ Luna pilot scored 16/16 in both arms: no measured uplift and no promotion. MiMo results are not part of this implementation claim.

CodeRabbit review is not requested during this early prototype, following the repository PR template.

Closes #3.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add frozen benchmark comparisons and evidence-backed candidate refinement

1 participant