feat(cli): eval group: move the evaluation commands under eval (#94 M1) by napetrov · Pull Request #325 · intel/dokimi

napetrov · 2026-10-31T06:25:51Z

Summary

Stage M1 of #94 (CLI API v2, tracking #276): the evaluation commands move under dokimi eval, handler-neutral. personas and questions roots go away; the evaluation leaves of tasks, catalog and repo move under eval, grouped into eval score, eval judge, eval questions, eval personas and eval triggers.

Every old path is a tombstone: it prints moved: dokimi <new argv> to stderr, exits 2 and runs nothing. Arguments carry over unchanged. repo run, repo report, repo compare and repo tasks replay stay where they are.

     Before                          After
tasks inspect                    eval inspect
repo tasks check                 eval check
tasks list                       eval tasks list
personas discover                eval personas discover
personas approve                 eval personas validate
questions generate               eval questions generate
questions analyze                eval questions analyze
questions refine                 eval questions refine
questions panel-review           eval questions review
questions import                 eval import
catalog trigger-set              eval triggers generate
eval score                       eval score pointwise
eval pairwise                    eval score pairwise
eval checklist                   eval score checklist
eval panel-score                 eval score panel
eval grounding                   eval score grounding
eval ragas                       eval score ragas
eval reference                   eval score reference
eval judge-validity              eval judge validate
eval judge-wording               eval judge wording
eval cross-judge                 eval judge agreement
personas (bare, or its help)     eval personas
questions (bare, or its help)    eval questions

Changes beyond the moves:

task_inspect.v1 consumers enum keeps the old labels and adds the new ones (removing enum values is breaking under schema_compat).
questions panel-review parser renamed to review to match the route table.
repo-eval.yml: the step runs dokimi eval check; the job name repo tasks check ${{ matrix.task }} is unchanged in case it is a required check.
Rebased onto intel/main 89809a25 (includes feat(repo): Codex and Copilot CLI harness profiles, native review mode #116 and fix(repo): scan accepts an Action without @ref that a workflow uses #306). Two docs files that feat(repo): Codex and Copilot CLI harness profiles, native review mode #116 touched were updated for the new names.
docs/reference/cli-interface.json regenerated (make cli-snapshot).

Evidence

tests/test_cli_eval_moves.py: for each of the 21 old paths and the two retired roots, exit 2 with moved: dokimi <new argv>, arguments preserved; each new path is a live leaf; the personas and questions roots are gone.
tests/test_cli_routes.py: tombstone table is clean (tombstone_problems() == []); paid leaves match the route table.
Full pytest: 11368 passed, 7 skipped (before the rebase onto 89809a25).
make ci-local with TMPDIR=/tmp on the final tree before the last rebase: 0 failed, 30 passed, 7 skipped (pytest inside: 11790 passed, 6 skipped). After the rebase, the diff is content-identical except index hashes and hunk offsets; the targeted tests (test_record_readers, test_judge_reliability, test_cli_eval_moves, test_cli_routes, test_repo_l0_cases, test_repo_l0_versions) pass: 572 passed. The PR's CI run is the full check of the rebased head.
Self-review (local mode): 4 passes (3 full medium rounds plus a delta recheck), blocking 0 on the final round. The L3 cap is 2 passes, so this is over budget. The plan's design lens and strongest-model verifier were not dispatched; probes were manual.
changelog.d/20261030_160000_napetrov_eval-group-m1.md: ### Changed with the before-to-after table and upgrade notes.

Merge Danger

CLI break, by design. Scripts that call an old path now fail with exit 2. The CLI: move commands into the new groups, stage M1–M3 (#94) #276 text says "no deprecation aliases"; this follows CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94 D4 (tombstones that name the new command, never run). Reviewers: confirm D4 still holds.
Name collisions. eval score is now a group (its old pointwise behaviour moved to eval score pointwise); moved_argv skips a tombstone when argv continues into a live leaf, and tombstone_problems allows a group to stay under its own name.
task_inspect output values change. consumers now emits eval triggers generate and eval import where it said catalog trigger-set and questions import. The schema enum keeps both, so the schema check passes, but a JSON consumer that matches the old strings must switch. Reviewers: confirm this is wanted rather than keeping the old emitted labels.
Required check name. The repo-eval job name is unchanged on purpose; check whether it is a required status before renaming it later.

Held or out of scope (flagged, not changed here)

Held for their own M-step: repo validate, explore list, native-import naming.
Researcher renames, catalog trigger-set --format to --dialect, and the python cli.py hints in baseline.py are out of scope.
Left unchanged on purpose: .agent-eval/tasks/review-pr-conventions/oracle_answer.md and review.patch (historical fixtures), and CHANGELOG.md and existing changelog.d/ fragments (history).

Refs #94, #276

🤖 Generated with Claude Code

M1) The evaluation commands now live under `dokimi eval`: personas, questions, the eval leaves of tasks, catalog and repo, and the `eval` leaves grouped into score, judge, questions, personas and triggers. The command functions are unchanged; the parsers and their registration move. `questions panel-review` is now `questions review`. Every old path is a tombstone: it prints `moved: dokimi <new argv>` to stderr and exits 2 without running. The table is in cli_routes.TOMBSTONES and tests/test_cli_eval_moves.py covers all 21 moves. Refs intel#94, intel#276 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

github-actions · 2026-10-31T06:26:59Z

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

- removed command `dokimi catalog trigger-set`
    - option `--catalog`
    - option `--format`
    - option `--help/-h`
    - option `--model`
    - option `--out`
    - option `--provider`
    - option `--questions`
    - option `--rewrite`
    - option `--strict`
    - option `--target`
    - option `--trigger-set`
- removed command `dokimi eval checklist`
    - option `--answers`
    - option `--arms`
    - option `--baseline-arm`
    - option `--concurrency`
    - option `--help/-h`
    - option `--judge-model`
    - option `--judge-provider`
    - option `--judge-temperature`
    - option `--output`
    - option `--questions`
- removed command `dokimi eval cross-judge`
    - option `--answers`
    - option `--baseline-arm`
    - option `--both`
    - option `--checklist`
    - option `--concurrency`
    - option `--grounded`
    - option `--help/-h`
    - option `--judge-temperature`
    - option `--judges`
    - option `--no-stored`
    - option `--output`
    - option `--pairwise`
    - option `--product`
    - option `--questions`
    - option `--require-validated`
    - option `--strict`
    - option `--validity`
- removed command `dokimi eval grounding`
    - option `--answers`
    - option `--help/-h`
    - option `--output`
    - option `--threshold`
- removed command `dokimi eval judge-validity`
    - option `--answers`
    - option `--bases`
    - option `--concurrency`
    - option `--context`
    - option `--escalate-model`
    - option `--escalate-provider`
    - option `--escalate-ties`
    - option `--help/-h`
    - option `--judge-model`
    - option `--judge-provider`
    - option `--judge-temperature`
    - option `--max-bases`
    - option `--min-accuracy`
    - option `--mode`
    - option `--output`
    - option `--product`
    - option `--scorer`
    - option `--seed`
    - option `--sequential`
    - option `--strict`
    - option `--typed-max-len`
    - option `--typed-threads`
    - option `--variants`
- removed command `dokimi eval judge-wording`
    - option `--answers`
    - option `--bases`
    - option `--concurrency`
    - option `--help/-h`
    - option `--judge-model`
    - option `--judge-provider`
    - option `--judge-temperature`
    - option `--min-accuracy`
    - option `--output`
    - option `--product`
    - option `--redraws`
    - option `--seed`
    - option `--strict`
    - option `--variants`
    - option `--wordings`
- removed command `dokimi eval pairwise`
    - option `--answers`
    - option `--arms`
    - option `--baseline-arm`
    - option `--concurrency`
    - option `--escalate-model`
    - option `--escalate-provider`
    - option `--escalate-ties`
    - option `--help/-h`
    - option `--judge-model`
    - option `--judge-provider`
    - option `--judge-temperature`
    - option `--output`
    - option `--product`
- removed command `dokimi eval panel-score`
    - option `--answers`
    - option `--concurrency`
    - option `--help/-h`
    - option `--judge-configs`
    - option `--judge-temperature`
    - option `--limit`
    - option `--model`
    - option `--output`
    - option `--product`
    - option `--provider`
    - option `--roles`
- removed command `dokimi eval ragas`
    - option `--answers`
    - option `--help/-h`
    - option `--ragas-model`
    - option `--ragas-provider`
- removed command `dokimi eval reference`
    - option `--answers`
    - option `--concurrency`
    - option `--help/-h`
    - option `--judge-model`
    - option `--judge-provider`
    - option `--no-judge`
    - option `--output/-o`
    - option `--questions`
- removed command `dokimi personas`
    - option `--help/-h`
- removed command `dokimi personas approve`
    - option `--file`
    - option `--help/-h`
- removed command `dokimi personas discover`
    - option `--count`
    - option `--description`
    - option `--github-token`
    - option `--help/-h`
    - option `--model`
    - option `--output`
    - option `--product`
    - option `--provider`
    - option `--repo`
    - option `--save-analysis`
- removed command `dokimi questions`
    - option `--help/-h`
- removed command `dokimi questions analyze`
    - option `--concurrency`
    - option `--help/-h`
    - option `--model`
    - option `--output`
    - option `--product`
    - option `--provider`
    - option `--questions`
- removed command `dokimi questions generate`
    - option `--context7-id`
    - option `--count`
    - option `--doc-source`
    - option `--embeddings`
    - option `--help/-h`
    - option `--model`
    - option `--output`
    - option `--personas`
    - option `--product`
    - option `--provider`
    - option `--top-k`
    - option `--topics`
    - option `--validate`
- removed command `dokimi questions import`
    - argument `path`
    - option `--format`
    - option `--grader`
    - option `--help/-h`
    - option `--output/-o`
    - option `--scorer`
    - option `--task`
- removed command `dokimi questions panel-review`
    - option `--concurrency`
    - option `--help/-h`
    - option `--limit`
    - option `--model`
    - option `--output`
    - option `--product`
    - option `--provider`
    - option `--questions`
    - option `--reviewers`
- removed command `dokimi questions refine`
    - option `--dry-run`
    - option `--fill-gaps`
    - option `--fill-model`
    - option `--fill-provider`
    - option `--help/-h`
    - option `--output`
    - option `--product`
    - option `--questions`
    - option `--sim-threshold`
    - option `--target-advanced`
    - option `--target-beginner`
    - option `--target-intermediate`
- removed command `dokimi repo tasks check`
    - argument `source`
    - option `--allow-unsupported`
    - option `--arm`
    - option `--build-network`
    - option `--build-timeout`
    - option `--cache`
    - option `--cache-dir`
    - option `--cache-root`
    - option `--docker`
    - option `--grade-timeout`
    - option `--help/-h`
    - option `--image`
    - option `--jobs/-j`
    - option `--manifest`
    - option `--manifest-file`
    - option `--out`
    - option `--rebuild-image`
    - option `--source-dir`
    - option `--task`
    - option `--warm`
- removed command `dokimi tasks inspect`
    - argument `path`
    - option `--help/-h`
    - option `--json`
    - option `--out`
- removed command `dokimi tasks list`
    - option `--help/-h`
    - option `--tasks-root`
+ added command `dokimi eval check`
    + argument `source`
    + option `--allow-unsupported`
    + option `--arm`
    + option `--build-network`
    + option `--build-timeout`
    + option `--cache`
    + option `--cache-dir`
    + option `--cache-root`
    + option `--docker`
    + option `--grade-timeout`
    + option `--help/-h`
    + option `--image`
    + option `--jobs/-j`
    + option `--manifest`
    + option `--manifest-file`
    + option `--out`
    + option `--rebuild-image`
    + option `--source-dir`
    + option `--task`
    + option `--warm`
+ added command `dokimi eval import`
    + argument `path`
    + option `--format`
    + option `--grader`
    + option `--help/-h`
    + option `--output/-o`
    + option `--scorer`
    + option `--task`
+ added command `dokimi eval inspect`
    + argument `path`
    + option `--help/-h`
    + option `--json`
    + option `--out`
+ added command `dokimi eval judge`
    + option `--help/-h`
+ added command `dokimi eval judge agreement`
    + option `--answers`
    + option `--baseline-arm`
    + option `--both`
    + option `--checklist`
    + option `--concurrency`
    + option `--grounded`
    + option `--help/-h`
    + option `--judge-temperature`
    + option `--judges`
    + option `--no-stored`
    + option `--output`
    + option `--pairwise`
    + option `--product`
    + option `--questions`
    + option `--require-validated`
    + option `--strict`
    + option `--validity`
+ added command `dokimi eval judge validate`
    + option `--answers`
    + option `--bases`
    + option `--concurrency`
    + option `--context`
    + option `--escalate-model`
    + option `--escalate-provider`
    + option `--escalate-ties`
    + option `--help/-h`
    + option `--judge-model`
    + option `--judge-provider`
    + option `--judge-temperature`
    + option `--max-bases`
    + option `--min-accuracy`
    + option `--mode`
    + option `--output`
    + option `--product`
    + option `--scorer`
    + option `--seed`
    + option `--sequential`
    + option `--strict`
    + option `--typed-max-len`
    + option `--typed-threads`
    + option `--variants`
+ added command `dokimi eval judge wording`
    + option `--answers`
    + option `--bases`
    + option `--concurrency`
    + option `--help/-h`
    + option `--judge-model`
    + option `--judge-provider`
    + option `--judge-temperature`
    + option `--min-accuracy`
    + option `--output`
    + option `--product`
    + option `--redraws`
    + option `--seed`
    + option `--strict`
    + option `--variants`
    + option `--wordings`
+ added command `dokimi eval personas`
    + option `--help/-h`
+ added command `dokimi eval personas discover`
    + option `--count`
    + option `--description`
    + option `--github-token`
    + option `--help/-h`
    + option `--model`
    + option `--output`
    + option `--product`
    + option `--provider`
    + option `--repo`
    + option `--save-analysis`
+ added command `dokimi eval personas validate`
    + option `--file`
    + option `--help/-h`
+ added command `dokimi eval questions`
    + option `--help/-h`
+ added command `dokimi eval questions analyze`
    + option `--concurrency`
    + option `--help/-h`
    + option `--model`
    + option `--output`
    + option `--product`
    + option `--provider`
    + option `--questions`
+ added command `dokimi eval questions generate`
    + option `--context7-id`
    + option `--count`
    + option `--doc-source`
    + option `--embeddings`
    + option `--help/-h`
    + option `--model`
    + option `--output`
    + option `--personas`
    + option `--product`
    + option `--provider`
    + option `--top-k`
    + option `--topics`
    + option `--validate`
+ added command `dokimi eval questions refine`
    + option `--dry-run`
    + option `--fill-gaps`
    + option `--fill-model`
    + option `--fill-provider`
    + option `--help/-h`
    + option `--output`
    + option `--product`
    + option `--questions`
    + option `--sim-threshold`
    + option `--target-advanced`
    + option `--target-beginner`
    + option `--target-intermediate`
+ added command `dokimi eval questions review`
    + option `--concurrency`
    + option `--help/-h`
    + option `--limit`
    + option `--model`
    + option `--output`
    + option `--product`
    + option `--provider`
    + option `--questions`
    + option `--reviewers`
+ added command `dokimi eval score checklist`
    + option `--answers`
    + option `--arms`
    + option `--baseline-arm`
    + option `--concurrency`
    + option `--help/-h`
    + option `--judge-model`
    + option `--judge-provider`
    + option `--judge-temperature`
    + option `--output`
    + option `--questions`
+ added command `dokimi eval score grounding`
    + option `--answers`
    + option `--help/-h`
    + option `--output`
    + option `--threshold`
+ added command `dokimi eval score pairwise`
    + option `--answers`
    + option `--arms`
    + option `--baseline-arm`
    + option `--concurrency`
    + option `--escalate-model`
    + option `--escalate-provider`
    + option `--escalate-ties`
    + option `--help/-h`
    + option `--judge-model`
    + option `--judge-provider`
    + option `--judge-temperature`
    + option `--output`
    + option `--product`
+ added command `dokimi eval score panel`
    + option `--answers`
    + option `--concurrency`
    + option `--help/-h`
    + option `--judge-configs`
    + option `--judge-temperature`
    + option `--limit`
    + option `--model`
    + option `--output`
    + option `--product`
    + option `--provider`
    + option `--roles`
+ added command `dokimi eval score pointwise`
    + option `--answers`
    + option `--concurrency`
    + option `--help/-h`
    + option `--judge-model`
    + option `--judge-provider`
    + option `--judge-temperature`
    + option `--output`
    + option `--product`
    + option `--run-id`
+ added command `dokimi eval score ragas`
    + option `--answers`
    + option `--help/-h`
    + option `--ragas-model`
    + option `--ragas-provider`
+ added command `dokimi eval score reference`
    + option `--answers`
    + option `--concurrency`
    + option `--help/-h`
    + option `--judge-model`
    + option `--judge-provider`
    + option `--no-judge`
    + option `--output/-o`
    + option `--questions`
+ added command `dokimi eval tasks`
    + option `--help/-h`
+ added command `dokimi eval tasks list`
    + option `--help/-h`
    + option `--tasks-root`
+ added command `dokimi eval triggers`
    + option `--help/-h`
+ added command `dokimi eval triggers generate`
    + option `--catalog`
    + option `--format`
    + option `--help/-h`
    + option `--model`
    + option `--out`
    + option `--provider`
    + option `--questions`
    + option `--rewrite`
    + option `--strict`
    + option `--target`
    + option `--trigger-set`
~ `dokimi eval score` subcommand required: None -> True
~ `dokimi eval score` parameters changed:
    - removed option `--answers`
    - removed option `--concurrency`
    - removed option `--judge-model`
    - removed option `--judge-provider`
    - removed option `--judge-temperature`
    - removed option `--output`
    - removed option `--product`
    - removed option `--run-id`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

exai-bot · 2026-10-31T06:30:31Z

✅ OpenCodeReview: Review complete: 0 finding(s) across 116 selected item(s).

Review details

Commit reviewed: d58f125 against merge base b4a0a8c
Coverage: 116 files in 18 groups
Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
Usage: 2.13M tokens (1.20M from cache), 45s
Tool calls: 134 (96 file_read, 29 code_search, 9 file_read_diff)
Build and CI configuration: 5 files
Project overview and contributor documentation: 8 files
CLI routing, interface contract, and command migration: 8 files
Bundled evaluation skills and selection: 4 files
Evaluation methodology and operating guides: 6 files
Repository evaluation and review documentation: 5 files
User journeys and outcome tracing: 5 files
Answer formats and answer imports: 4 files
Catalog command and tests: 2 files
Treatment arms and persona workflows: 6 files
Question generation, schemas, and validation: 10 files
Evaluation execution, reference checks, and output errors: 8 files
Judge reliability, validity, and cross-judge behavior: 10 files
Model providers, defaults, and call budgets: 7 files
Repository commands, hosting, and run lifecycle: 9 files
Repository grading, task selection, and session execution: 3 files
Task format model, inspection, and corpus tests: 8 files
External content ingestion and record readers: 8 files

codecov-commenter · 2026-10-31T06:36:03Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.05%. Comparing base (b4a0a8c) to head (d58f125).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #325   +/-   ##
=======================================
  Coverage   94.05%   94.05%           
=======================================
  Files         297      297           
  Lines       38018    38033   +15     
  Branches     6424     6426    +2     
=======================================
+ Hits        35756    35771   +15     
- Misses       1452     1453    +1     
+ Partials      810      809    -1

Files with missing lines	Coverage Δ
dokimi/dokimi/caching.py	`98.71% <ø> (ø)`
dokimi/dokimi/cli_routes.py	`100.00% <100.00%> (ø)`
dokimi/dokimi/commands/answer_formats.py	`65.90% <100.00%> (ø)`
dokimi/dokimi/commands/answers.py	`95.23% <ø> (ø)`
dokimi/dokimi/commands/arms.py	`97.46% <ø> (ø)`
dokimi/dokimi/commands/catalog.py	`97.82% <100.00%> (+<0.01%)`	⬆️
dokimi/dokimi/commands/evaluate.py	`90.09% <100.00%> (+0.15%)`	⬆️
dokimi/dokimi/commands/judge_reliability.py	`93.59% <100.00%> (ø)`
dokimi/dokimi/commands/personas.py	`89.58% <100.00%> (ø)`
dokimi/dokimi/commands/questions.py	`78.62% <100.00%> (ø)`
... and 18 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…m1-eval Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/docs/benchmarking-and-comparison.md

The top-level CLI table still named personas and questions as live roots. Both are tombstones now (they exit 2 with a moved line), so list them as eval personas and eval questions. Thread: PRRT_kwDOVBg7g86yOm97 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

eval tasks list --tasks-root <missing> is refused with exit 2 and a "not a directory" line, not a silent success. The empty-listing trap applies only to an existing empty directory. Thread: PRRT_kwDOVBg7g86yOm9z Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

eval score is a live group, so the moved-and-exit-2 rule does not cover bare eval score: it exits 2 with a usage error, and eval score --help lists the leaves. Thread: PRRT_kwDOVBg7g86yOm9x Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

The stale bare eval score reference in the methodology (judge-independence warning), benchmarking and runs docs now names eval score pointwise. Surface lists that named the tombstoned questions and personas roots now name eval questions and eval personas. Thread: PRRT_kwDOVBg7g86yOm94 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

The new "see eval inspect" line in import_path had no test (Codecov patch: 1 line missing). A file no adapter recognises now has a test that names the error and the path. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

exai-bot · 2026-10-31T19:37:39Z

  hits, so on those queries it is stricter than the `skill-catalog:` arm; `skill-creator --target <skill>` (a catalog skill) writes an
  [anthropics/skills skill-creator](https://github.com/anthropics/skills/tree/main/skills/skill-creator)
-  eval set; `questions` writes `questions.v2` with `expected_skill` for the arm below.
+  eval set; `eval questions` writes `questions.v2` with `expected_skill` for the arm below.


Nit: questions here is an output-format value, not the retired CLI root. eval triggers generate still accepts --format questions (alongside reach and skill-creator); --format 'eval questions' is invalid, and the eval questions group does not perform this export. Keep the format name unchanged, preferably spelling it as --format questions.

Suggestion:

Suggested change

eval set; `eval questions` writes `questions.v2` with `expected_skill` for the arm below.

eval set; `--format questions` writes `questions.v2` with `expected_skill` for the arm below.

Right, and thanks for the catch. eval questions has only the generate, analyze, refine and review leaves (dokimi eval questions --help), so it does not write questions.v2 with expected_skill. The export is eval triggers generate --format questions: --format {reach,skill-creator,questions} is in its help, and catalog.py _write_trigger_set calls save_questions_artifact(ts.to_questions(...)) for it.

Fixed in d58f125 with the line spelled as you suggested, plus the full command name so the flag cannot be read as belonging to the bullet's skill-creator item.

…s group `eval questions` has generate, analyze, refine and review leaves only; the questions.v2 export with expected_skill is `eval triggers generate --format questions`. Thread: PRRT_kwDOVBg7g86yT_9o Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

vlad-perevezentsev

Review at d58f125. The move itself is right. Each of the 21 old paths prints moved: dokimi <new argv> and exits 2 without running. Each new path is a leaf with the same options as the old one. No code, test, bundled skill or workflow still calls an old path. What is left is wording (help text, the changelog, a few docs) and one routing case: a typo after eval score. Both are in the line comments.

What I ran, at d58f125:

make ci-local --offline: 0 failed, 19 passed, 13 skipped (the network and Docker steps).
The CLI snapshot of the base against this head: the only leaves removed are the 21 old paths, and the only leaves added are their 21 new paths. For each pair the options are the same (names, defaults, choices, required). No other leaf changed.
A mutation run: I changed the tombstone exit code from 2 to 97. Only the 50 tombstone tests in test_cli_eval_moves.py and test_cli_routes.py fail; 12,002 pass. So no other test still calls an old path and passes only because that path exits 2.
The old and new paths by hand: eval score bare, with --help, with --answers a.json and with a typo; personas --help; repo tasks check .. I also parsed the repo-eval.yml step's argv (eval check . --manifest --task --out --warm --allow-unsupported): it reaches cmd_repo_tasks_check with the same flags.

Checked, no issues

Tombstones: they follow #94 D4 and its migration policy: name the new command, run nothing, exit 2. eval score followed by a leaf, bare, or with only --help is not caught by its own tombstone.
task_inspect.v1 labels: tasks inspect and this schema are not in 0.1.0, so no released output has the old consumers labels. The old enum values are kept only so that schema_compat against the base passes, and nothing in dokimi reads the labels back.
Dispatch: no code reads the subcommand dest names that changed, and every new group requires a leaf.
The bot's threads: all five are fixed at this head, including the open one on --format questions (evaluating-treatments.md:553).

vlad-perevezentsev · 2026-10-31T23:15:23Z

+        # A group that stays under its own name (`eval score` -> `eval score pointwise`) keeps its
+        # help and its usage error when it has no leaf; any other argument is an old call.
+        if tomb.new[:n] == _canonical(tomb.old) and all(a in ("-h", "--help") for a in argv[n:]):
+            continue


A typo after eval score is reported as a moved command.

dokimi eval score pointwse --answers a prints moved: dokimi eval score pointwise pointwse --answers a and exits 2. eval score data.json does the same. The old eval score took only options, so a plain word right after eval score is never an old call, and the suggested argv is not a command either. argparse's own invalid choice: 'pointwse' (choose from pointwise, ragas, ...) is the useful message here.

Suggestion: skip a group that stays in place unless the next argument is an option:

if tomb.new[:n] == _canonical(tomb.old) and ( all(a in ("-h", "--help") for a in argv[n:]) or not argv[n].startswith("-") ): continue

I tried this. test_cli_eval_moves.py and test_cli_routes.py pass, except test_an_old_path_exits_2_and_names_the_new_argv[eval score], which feeds eval score data.json. Give that test an option tail (--answers a.json) instead.

vlad-perevezentsev · 2026-10-31T23:15:23Z

+
+  | Before | After |
+  |---|---|
+  | `tasks inspect` | `eval inspect` |


33 other fragments in changelog.d/ still name the old paths, and they go into the same release notes as this one.

The PR description calls them history, but they have not been released yet. scriv collect renders every file in changelog.d/ into the next release and only then deletes it; 0.1.0 is already in CHANGELOG.md. So 0.2.0 will list "Added: tasks inspect <path>" (20261023_120000_napetrov_task-format-detect.md) and, in the same notes, say that tasks inspect moved. tasks inspect, questions import and eval reference are not in 0.1.0 at all. For someone upgrading, they are new commands announced under names that already exit 2.

Suggestion: rename the paths in the unreleased fragments using the table above. At least do it in the eight that mention tasks inspect, questions import or eval reference.

vlad-perevezentsev · 2026-10-31T23:15:23Z

-def register(sub, positive_int) -> None:
-    """Add the `repo` subcommand group."""
+def register(sub, positive_int, eval_sub) -> None:
+    """Add the `repo` subcommand group. Its check moved to `eval check` (#94 M1)."""


The repo help still lists the "task self-check", which moved to eval check.

dokimi --help and dokimi repo --help show "... budget, task self-check, agent runs and their report" (line 429), but repo tasks check now only prints moved: dokimi eval check.

Suggestion: drop "task self-check" from that help string.

vlad-perevezentsev · 2026-10-31T23:15:23Z

-def register(sub, positive_int) -> None:
-    """Add the `eval` subcommand group."""
+def register(sub, positive_int):
+    """Add the `eval` subcommand group and its `score` and `judge` groups; return the eval subparsers.


The eval help no longer says what the group holds.

dokimi --help shows eval Evaluate answers [paid] (line 471). The README's CLI block says "Score answers, run RAGAS, or use a multi-judge panel" (README.md:128). eval now also holds personas, questions, import, inspect, check, tasks and triggers. So someone looking for where questions or repo tasks check went gets no hint from the top-level help.

Suggestion: something like help="Build and check eval sets, score answers, calibrate judges", and the same in the README line.

vlad-perevezentsev · 2026-10-31T23:15:23Z

 scores degraded answers lower.

-![Three checks of an eval set: valid (dokimi repo validate), graded right (dokimi repo tasks check), judge checked (dokimi eval judge-validity)](assets/readme/eval-checks.png)
+![Three checks of an eval set: valid (dokimi repo validate), graded right (dokimi eval check), judge checked (dokimi eval judge validate)](assets/readme/eval-checks.png)


The picture still shows the old commands.

The alt text now says dokimi eval check and dokimi eval judge validate. The image assets/readme/eval-checks.png itself still shows $ dokimi repo tasks check and $ dokimi eval judge-validity, and both of these now exit 2.

Suggestion: regenerate the image with the new names.

vlad-perevezentsev · 2026-10-31T23:15:23Z

 | `catalog scan --root <missing>` | `--root is not a directory: <path>` | 1 |
 | `tasks run --tasks-root <missing>` | `FileNotFoundError: Task directory not found: demo-widget` | 1 |
-| `tasks list --tasks-root <missing>` | prints nothing | 0 |
+| `eval tasks list --tasks-root <missing>` | `--tasks-root is not a directory: <path>` | 2 |


Nit: The two rows above have the same problem the bot found in this one.

On this head, catalog scan --root /nonexistent exits 2, not 1. tasks run --tasks-root /nonexistent prints --tasks-root is not a directory: /nonexistent and also exits 2.

Suggestion:

| `catalog scan --root <missing>` | `--root is not a directory: <path>` | 2 | | `tasks run --tasks-root <missing>` | `--tasks-root is not a directory: <path>` | 2 |

vlad-perevezentsev · 2026-10-31T23:15:23Z

      # Builds the image from the manifest's Dockerfile (the tree without its
      # tasks), then grades the null and the oracle workspace in it.
-      - name: repo tasks check
+      - name: eval check


Nit: The step now runs eval check, but the header comment still says the job runs repo tasks check (line 11).

The Summary step also prints ### repo tasks check headings (lines 411 and 415). If those headings follow the job name on purpose, keep them. The header comment describes the command, so it should name eval check.

vlad-perevezentsev · 2026-10-31T23:15:24Z

+    A path that is itself a live leaf under the old prefix is not a tombstone: ``eval score
+    pointwise`` is live even though ``eval score`` moved there.


Nit: Three docstrings here no longer match the code.

Leaf (line 39) gives ("eval", "score") as its leaf example, but that is now a group. Tombstone (line 48) says it always answers with new and exits 2, which is not true for a bare eval score or for eval score --help. _is_route (line 265) says a tombstone on a group shadows every route under it, but moved_argv now skips live leaves. These docstrings explain the tricky part for whoever writes the M2/M3 entries.

Suggestion: use ("eval", "score", "pointwise") as the example, and state the exception for a group that stays in place in the other two.

vlad-perevezentsev · 2026-10-31T23:15:24Z

+    tasks_p = repo_sub.add_parser("tasks", help="Review tasks built from past reviews (no LLM)")
    tasks_sub = tasks_p.add_subparsers(dest="repo_tasks_cmd", required=True)
-    check_p = tasks_sub.add_parser(
+    # eval check (#94 M1): the self-check of `eval check`, moved under eval


Nit: A few old names are left, one of them in user output.

evaluate.py:146 and :198 pass context="panel-score" / "eval-score", so eval score panel prints ⚠ Evaluator independence warning (panel-score): ... to stderr.

questions.v2.json:225 still says questions generate --validate.

This comment says "the self-check of eval check, moved under eval", which is circular: it moved from repo tasks check. Line 26 still says "shared by tasks check and run".

The module docstring at evaluate.py:4 still says "score, ragas, panel-score".

test_cli_routes.py:267 still says "no command moves in this release".

vlad-perevezentsev · 2026-10-31T23:15:24Z

 # Build image for dokimi's own repository-mode evals: CI's Python, the lock, every CI tool,
-# and a warm uv cache so `make dev` inside a run works without downloading anything.
-# Pinned by digest, like the tools below, so a rerun grades in the same image. To move it:
-# `docker buildx imagetools inspect python:3.11-slim-bookworm`, then `repo tasks check`.
+# pinned by digest. To move the pin: `docker buildx imagetools inspect python:3.11-slim-bookworm`, then `eval check`.


Nit: Shortening this comment dropped the reason for the warm uv cache.

The old text said the cache is there so make dev inside a run downloads nothing. Lines 18-21 now warm it with no word on why, so someone bumping the image may drop it.

Suggestion: put a one-line comment above line 18, which keeps the two-line rule:

# A warm uv cache, so `make dev` inside a run downloads nothing.

napetrov requested a review from a team as a code owner October 31, 2026 06:25

napetrov added the cli-interface-change Changes dokimi's user-facing CLI surface label Oct 31, 2026

exai-bot reviewed Oct 31, 2026

View reviewed changes

Comment thread dokimi/changelog.d/20261030_160000_napetrov_eval-group-m1.md Resolved

Comment thread dokimi/docs/external-content.md Resolved

Comment thread dokimi/docs/methodology.md Resolved

Comment thread dokimi/README.md Resolved

napetrov added 6 commits October 31, 2026 12:17

Merge branch 'main' of https://github.com/intel/dokimi into feat/cli-…

61a4863

…m1-eval Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/docs/benchmarking-and-comparison.md

exai-bot reviewed Oct 31, 2026

View reviewed changes

vlad-perevezentsev reviewed Oct 31, 2026

View reviewed changes

	eval set; `eval questions` writes `questions.v2` with `expected_skill` for the arm below.
	eval set; `--format questions` writes `questions.v2` with `expected_skill` for the arm below.

		A path that is itself a live leaf under the old prefix is not a tombstone: ``eval score
		pointwise`` is live even though ``eval score`` moved there.

Conversation

napetrov commented Oct 31, 2026

Summary

Evidence

Merge Danger

Held or out of scope (flagged, not changed here)

Uh oh!

github-actions Bot commented Oct 31, 2026

CLI interface change detected

Uh oh!

exai-bot commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vlad-perevezentsev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

exai-bot commented Oct 31, 2026 •

edited

Loading

codecov-commenter commented Oct 31, 2026 •

edited

Loading