feat(catalog): known-positive guard for catalog trigger-probe by napetrov · Pull Request #320 · intel/dokimi

napetrov · 2026-10-31T04:57:08Z

Summary

catalog trigger-probe now checks that the harness can load the skill before it scores the labelled set. A skill the harness never shows scores every query 0%, which reads as a finding.

 catalog trigger-probe --catalog <dir> --trigger-set <set> --out probe.json
+  known_positive(target or first skill)   # one run: "Use the <skill> skill."
+    loaded the skill   -> continue to the labelled queries   "status": "ok"
+    did not            -> write status "invalid: skill not visible" to --out and stdout,
+                          exit 1, no labelled query runs
   labelled queries ...                    (unchanged)
+  --no-known-positive                     skip the check      "status": "unchecked"

The output keeps every existing key; status and known_positive are added. The refusal costs one query. The check runs one skill, so it shows that the harness loads skills at all, not that every skill in the catalog is visible; not_listed still covers skills missing from the init listing.

Evidence

New tests in tests/test_trigger_probe.py use a fake runner and no network. Against the base sources they fail, and they pass on this branch.

before: harnesses/trigger_probe.py and commands/catalog_probe.py reverted to intel/main, tests kept
  7 failed, 23 deselected   (known_positive, guard refusal, --no-known-positive, target choice)

after (py3.13, uv run --extra dev pytest -n auto -q):
  11286 passed, 26 skipped
ruff check .: passed    mypy dokimi: no issues in 295 files
python -m dokimi.schema_check: ok    python -m dokimi.config_check: ok

ruff format --check is not enforced in this repo (the [tool.ruff] comment says so), and the touched files were already unformatted at the base, so the diff keeps the existing layout.

Merge Danger

Exit code: a catalog whose first (or --target) skill is not loaded now exits 1 before any labelled run, where it used to write a 0% report and exit 0. Scripts that expect a report must pass --no-known-positive or fix the catalog.
One extra paid run per invocation (a single explicit query).
A failed known-positive run (CLI cannot start, timeout before any Skill call) also reports invalid: skill not visible; known_positive.error and the stderr message carry the real cause.
New flag --no-known-positive is allowlisted in tests/data/cli_flags_allowlist.txt (no registry concept yet) and docs/reference/cli-interface.json is regenerated.

🤖 Generated with Claude Code

Before the labelled queries, one explicit run ('Use the <skill> skill.') must load that skill. If it does not, the probe writes status 'invalid: skill not visible', prints it, exits 1 and runs no labelled query, so a skill the harness never shows cannot read as a 0% trigger rate. --no-known-positive skips the check (status 'unchecked'). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

github-actions · 2026-10-31T04:58:22Z

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi catalog trigger-probe` parameters changed:
    + added option `--no-known-positive`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

exai-bot · 2026-10-31T05:01:16Z

✅ OpenCodeReview: Review complete: 0 finding(s) across 7 selected item(s).

Review details

Commit reviewed: 48cbd0f against merge base b4a0a8c
Coverage: 7 files in 1 groups
Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
Usage: 103k tokens (45k from cache), 23s
Tool calls: 11 (9 file_read, 2 code_search)
Known-positive guard for treatment probes, CLI configuration, tests, and documentation: 7 files

exai-bot · 2026-10-31T05:01:17Z

+        return None
+    from dokimi.harnesses import trigger_probe as tp
+
+    known = tp.known_positive(args.target or sorted(names)[0], catalog, cfg)


Must-fix: names can be empty here. For example, use an empty existing --catalog directory and a trigger set containing {"questions":[{"id":"a","question":"Hello","expected_skill":null}]} with no --target. Discovery returns no skills, and _labelled_queries accepts the explicitly labelled negative query because its acceptable-skill list is empty. The new default guard then raises IndexError at sorted(names)[0], instead of reporting a controlled input error. Reject an empty catalog before selecting the guard skill, and add a regression test for this negative-only set.

Fixed in 1375814. An empty --catalog is now refused with --catalog holds no skills before the known-positive guard picks its skill, so a negative-only set no longer reaches sorted(names)[0]. Runs with --no-known-positive are unchanged.

Regression tests: test_cli_refuses_an_empty_catalog_before_the_guard_picks_a_skill (red before the fix, IndexError at catalog_probe.py:123), and test_no_known_positive_still_runs_a_negative_only_set_on_an_empty_catalog.

codecov-commenter · 2026-10-31T05:12:40Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.04%. Comparing base (b4a0a8c) to head (48cbd0f).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #320      +/-   ##
==========================================
- Coverage   94.05%   94.04%   -0.01%     
==========================================
  Files         297      297              
  Lines       38018    38042      +24     
  Branches     6424     6427       +3     
==========================================
+ Hits        35756    35778      +22     
- Misses       1452     1454       +2     
  Partials      810      810

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/catalog_probe.py	`90.99% <100.00%> (+1.62%)`	⬆️
dokimi/dokimi/harnesses/trigger_probe.py	`90.25% <100.00%> (+0.46%)`	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…-positive-guard

A negative-only trigger set against a catalog with no skills reached sorted(names)[0] in the known-positive guard and raised IndexError. Refuse the empty catalog as an input error before the guard picks its skill. Runs with --no-known-positive are unchanged. Cover both with regression tests. Thread: PRRT_kwDOVBg7g86yOGu2 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

vlad-perevezentsev

Review at 48cbd0f. The guard does what it says on the happy path: it runs once before any labelled query, its cost is added to the total, and --no-known-positive keeps the old behaviour. The main gap is that a failed run (API error, timeout) is reported as "skill not visible"; the details are in the line comments.

What I ran, at 48cbd0f:

make ci-local --offline: 0 failed, 19 passed, 13 skipped (network/Docker jobs); the test suite has 11,980 passed, 58 skipped.
A fake claude that lists sum-fast in its init event and then returns an is_error result: the probe exits 1 with "status": "invalid: skill not visible".

Checked, no issues

Empty catalog (the bot's thread): fixed in 48cbd0f; an empty --catalog is refused before sorted(names)[0], and both new tests cover it.
Order of checks: bad input, a bad --out and an unlabelled set still exit before the paid guard run.
CLI snapshot and allowlist: --no-known-positive is in cli-interface.json and cli_flags_allowlist.txt; the changelog fragment is under Added.

vlad-perevezentsev · 2026-10-31T22:43:06Z

+    _write(out_path, {"harness": "claude-code", "status": _INVALID, "known_positive": known,
+                      "mode": "sealed" if args.sealed else "stock", "model": args.model,
+                      "catalog": str(catalog), "answers": []})
+    print(json.dumps({"status": _INVALID, "known_positive": known}, indent=2))
+    raise SystemExit(f"{_INVALID}: {known['query']!r} did not load {known['skill']!r}"
+                     + (f" ({known['error']})" if known["error"] else "") + "; no labelled query was run")


Any failure of the guard run is written as status: "invalid: skill not visible", including failures that say nothing about visibility.

A timeout, no result event, an API or auth error (result.is_error), or an OSError/RuntimeError from staging all reach this branch, because it only checks known["passed"]. Repro: a fake claude that prints an init event listing sum-fast and then {"type":"result","subtype":"error_during_execution","is_error":true}. The probe exits 1 and probe.json says "status": "invalid: skill not visible", with "error": "error_during_execution" inside known_positive, even though that same run listed the skill. Someone reading the artifact concludes the harness hides the skill when the run simply failed, which is the "an error reads as a result" case dokimi tries to avoid.

Suggestion: when known["error"] is set, write a separate status such as error: known-positive run failed (and say so in the SystemExit message and the docs); keep invalid: skill not visible for a run that finished without an error and did not load the skill. A test with an is_error result would cover it.

vlad-perevezentsev · 2026-10-31T22:43:06Z

+    query = f"Use the {skill} skill."
+    try:
+        r = run_query(query, catalog, cfg, runner=runner, source_env=source_env)
+    except (OSError, RuntimeError) as exc:
+        return {"skill": skill, "query": query, "passed": False, "loaded": [], "cost_usd": None,
+                "error": f"{type(exc).__name__}: {exc}"}
+    return {"skill": skill, "query": query, "passed": skill in r["skill"]["loaded"],
+            "loaded": r["skill"]["loaded"], "cost_usd": r["cost_usd"], "error": r["error"]}


The guard shows that the Skill tool accepts the name from the prompt, not that the harness listed or described the skill.

Use the <skill> skill. names the skill, so the model can call Skill(<skill>) even if the listing budget (--listing-budget-fraction) cut its entry or description. In that case the guard passes and the labelled queries can still score about 0% for the reason the docs say the guard rules out ("the harness never showed it", evaluating-treatments.md:603-604). The guard run already parses the init event, but known_positive drops r["skill"]["listed"].

Suggestion: keep listed in the known_positive record, and either require the skill to be listed when the init event lists any skills, or narrow the docs to "the skill can be loaded". Do you know whether Claude Code accepts a Skill call for a skill it did not list?

vlad-perevezentsev · 2026-10-31T22:43:06Z

+    _write(out_path, {"harness": "claude-code", "status": _INVALID, "known_positive": known,
+                      "mode": "sealed" if args.sealed else "stock", "model": args.model,
+                      "catalog": str(catalog), "answers": []})


The invalid output has no top-level cost_usd and drops header fields the normal output always has.

The guard run is paid, but its cost is only in known_positive.cost_usd; trials, max_turns, listing_budget_fraction, claude_code_version are missing too. A script that sums out["cost_usd"] over probe files raises KeyError or misses the spend. The header is also rebuilt by hand here, so it will keep drifting from the dict at lines 82-89.

Suggestion: build the shared header (harness, status, known_positive, mode, model, catalog, settings) in one helper used by both writes, and set cost_usd to the guard's cost in the invalid output.

vlad-perevezentsev · 2026-10-31T22:43:06Z

+    cost += (known or {}).get("cost_usd") or 0
    out = {
        "harness": "claude-code",
+        "status": "ok" if known else "unchecked",


Nit: status is ok whenever the guard passed, even when the same run then exits 1 because not_listed is non-empty.

The output then says "status": "ok" while the CLI raises catalog skills Claude Code did not list: .... A consumer that filters on status == "ok" accepts a run the command rejected.

Suggestion: either set a non-ok status when not_listed is non-empty, or name the field after what it reports (for example known_positive_status).

vlad-perevezentsev · 2026-10-31T22:43:06Z

-labelled query exits 2 before any run. Records are kept in `<out>.partial` while the
-probe runs:
+labelled query exits 2 before any run. Before the labelled queries, one explicit run
+(`Use the <skill> skill.`, for the `--target` skill or the first skill of the catalog) must


Nit: "The first skill of the catalog" is the alphabetically first frontmatter name (sorted(names)[0]), not the first directory or listing entry.

With sum-fast/ and io-read/ the guard checks io-read. Also, since 48cbd0f an empty --catalog is refused unless --no-known-positive is passed; before this PR a negative-only set ran on it. Neither the docs nor the changelog fragment say so.

Suggestion: write "the alphabetically first skill name" here and in the changelog fragment, and add a sentence that an empty catalog is refused while the guard is on.

napetrov requested a review from a team as a code owner October 31, 2026 04:57

github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Oct 31, 2026

exai-bot reviewed Oct 31, 2026

View reviewed changes

Merge branch 'main' of https://github.com/intel/dokimi into fix/known…

99d7347

…-positive-guard

napetrov force-pushed the fix/known-positive-guard branch from 1375814 to 48cbd0f Compare October 31, 2026 20:00

vlad-perevezentsev reviewed Oct 31, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(catalog): known-positive guard for catalog trigger-probe - #320

feat(catalog): known-positive guard for catalog trigger-probe#320
napetrov wants to merge 3 commits into
intel:mainfrom
napetrov:fix/known-positive-guard

napetrov commented Oct 31, 2026

Uh oh!

github-actions Bot commented Oct 31, 2026

Uh oh!

exai-bot commented Oct 31, 2026 •

edited

Loading

Uh oh!

exai-bot Oct 31, 2026

Uh oh!

napetrov Oct 31, 2026

Uh oh!

codecov-commenter commented Oct 31, 2026 •

edited

Loading

Uh oh!

vlad-perevezentsev left a comment

Uh oh!

vlad-perevezentsev Oct 31, 2026

Uh oh!

vlad-perevezentsev Oct 31, 2026

Uh oh!

vlad-perevezentsev Oct 31, 2026

Uh oh!

vlad-perevezentsev Oct 31, 2026

Uh oh!

vlad-perevezentsev Oct 31, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

napetrov commented Oct 31, 2026

Summary

Evidence

Merge Danger

Uh oh!

github-actions Bot commented Oct 31, 2026

CLI interface change detected

Uh oh!

exai-bot commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

exai-bot Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

napetrov Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vlad-perevezentsev left a comment

Choose a reason for hiding this comment

Uh oh!

vlad-perevezentsev Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

vlad-perevezentsev Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

vlad-perevezentsev Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

vlad-perevezentsev Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

vlad-perevezentsev Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

exai-bot commented Oct 31, 2026 •

edited

Loading

codecov-commenter commented Oct 31, 2026 •

edited

Loading