Skip to content

feat(repo): guidance:overlay arm and overlay reach check - #338

Open
napetrov wants to merge 7 commits into
intel:mainfrom
napetrov:ce-overlay
Open

napetrov wants to merge 7 commits into
intel:mainfrom
napetrov:ce-overlay

Conversation

@napetrov

@napetrov napetrov commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Refs agentsmd/agents.md#185.

Summary

A harness profile names its agent with a new overlay key. The new arm keeps the base and that agent's overlay, and drops the rest:

 repo/
   AGENTS.md
+  AGENTS.claude.md       # kept for claude-code (overlay: claude)
-  AGENTS.codex.md        # removed by guidance:overlay under claude-code
-  sub/AGENTS.copilot.md  # removed
guidance:overlay (profile.overlay = claude)
  keep  AGENTS.md, every AGENTS.claude.md (any depth)
  drop  every other AGENTS.<name>.md
  refuse if the profile has no overlay, the base has no AGENTS.claude.md,
           or a kept link names a removed overlay (the base would dangle)

dokimi repo reach now probes every AGENTS.<name>.md. A foreign overlay that a harness loaded is reported as a finding: line and in the overlays block of guidance_reach.v1 (cross_loaded). own_loaded says whether the harness loads its own overlay at all. The arm does not make a harness load its overlay; the probe measures that.

Evidence

  • make ci-local on the head commit: 0 failed, 30 passed, 7 skipped (docker, micromamba, latest-Python and push-only jobs).
  • tests/test_repo_overlay.py and tests/test_repo_reach.py: 122 passed offline. Includes the new regressions for a link reached through a directory link, a symlink loop in the base, a base link whose target a removed link also names, and a profile overlay value with a trailing newline.
  • Self-review (pr-review local mode, 3 passes): every confirmed blocker was fixed. The last fixes were verified by tests and ci-local, not by a fresh review pass. The review record is in the author's local scratch, not in this PR.

Merge Danger

Door: two-way. The arm is new and the profile key is optional; reverting removes both with no data migration.

Blast radius: low for existing runs.

  • The arm names are unchanged, and the run id (batch_ledger) does not include the new overlay key, so existing cell ids are unchanged.
  • dokimi repo reach now probes overlay files too. Repos with AGENTS.<name>.md files will run more reach scenarios and cost more. Repos without overlays are unaffected.
  • Schema changes are widening only (harness_profile.v1: optional overlay; guidance_reach.v1: optional overlays).

Declined items

  • A check_ancestors guard that lacks overlay names could report an ancestor overlay as a false cross-load. Not probed.
  • overlay_of folds case, so AGENTS.CODEX.md counts as codex. A case-sensitive harness would not load it. Kept on purpose; tests assert it.
  • Alias attribution: a foreign-named symlink to the base AGENTS.md is reported as cross-loaded when only the base content was read. The docstring says a link counts as its target's overlay; say if you want it narrowed to reads by name.
  • Prior-round dismissals and the reasons are in the review record, not repeated here.

Open questions for maintainers

  1. The run id (batch_ledger) does not include the profile's overlay value. Editing it on a profile can reuse cells from a resumed batch built under the old overlay. Including it would change every claude-code cell id, so this PR does not change it.
  2. Should the arm require a base AGENTS.md? docs/repo-evaluation-contract.md §3 says it keeps the base, but the code only requires an overlay.
  3. Should check_ancestors also guard AGENTS.<name>.md files above the workspace?

🤖 Generated with Claude Code

A harness profile names its agent with an `overlay` key (claude-code:
claude, codex, copilot). The guidance:overlay arm keeps the base AGENTS.md
and the AGENTS.<overlay>.md of that agent, and removes every other
AGENTS.<name>.md. The arm is refused without a profile overlay and when the
base has no overlay for the agent.

The reach probe now probes every overlay, even where the manifest globs miss
it, and reports a foreign overlay that a harness loaded as a cross_loaded
finding. own_loaded says whether the harness loads its own overlay at all.

Refs agentsmd/agents.md#185

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Copilot AI balanced review requested due to automatic review settings November 1, 2026 02:09
@napetrov
napetrov requested a review from a team as a code owner November 1, 2026 02:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical symlink-validation and stale-inventory bugs can leave AGENTS.md dangling, retain foreign overlays, or cause cleanup to fail with FileNotFoundError.

2 open issues
What changed in this PR

2 new must-fix findings.

This PR adds per-agent overlay evaluation to dokimi’s repository workflow, allowing runs to isolate a harness’s own guidance and measure whether it reads other agents’ overlays.

Changes:

  • Adds guidance:overlay with profile-based selection and dangling-link checks.
  • Extends reach probing and reports with own-overlay and foreign-overlay loading results.
  • Updates built-in profiles, schemas, documentation, and regression tests.
File Description
dokimi/​tests/​test_repo_overlay.py Tests overlay filtering, validation, links, and reporting.
dokimi/​dokimi/​schemas/​harness_profile.v1.json Adds the optional overlay profile key.
dokimi/​dokimi/​schemas/​guidance_reach.v1.json Adds overlay reach results.
dokimi/​dokimi/​repos/​workspace.py Implements overlay workspace preparation.
dokimi/​dokimi/​repos/​reach.py Probes overlays and reports cross-loading.
dokimi/​dokimi/​repos/​profiles/​copilot.yaml Selects the Copilot overlay.
dokimi/​dokimi/​repos/​profiles/​copilot-review.yaml Selects the Copilot review overlay.
dokimi/​dokimi/​repos/​profiles/​codex.yaml Selects the Codex overlay.
dokimi/​dokimi/​repos/​profiles/​codex-review.yaml Selects the Codex review overlay.
dokimi/​dokimi/​repos/​profiles/​claude-code.yaml Selects the Claude overlay.
dokimi/​dokimi/​repos/​overlays.py Centralizes overlay filename recognition.
dokimi/​dokimi/​repos/​harness_profile.py Parses and validates overlay names.
dokimi/​dokimi/​commands/​repo_run.py Documents the new arm in CLI help.
dokimi/​docs/​repo-evaluation-contract.md Defines overlay arm and reach behavior.
dokimi/​docs/​preparing-a-repository.md Adds overlay evaluation guidance.
dokimi/​changelog.d/​20261031_120000_napetrov_ce-overlay.md Records the new arm and reach reporting.

Review effort: Balanced

Comment thread dokimi/dokimi/repos/workspace.py Outdated
Comment on lines +463 to +464
if arm.base == OVERLAY_ARM:
return _apply_overlay(ws, tracked, overlay or "")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ee38237. The overlay inventory now comes from the workspace (git ls-files --cached --others, existing paths only), not the snapshot index, so an overlay added by a setup patch is seen. Removal is checked against that inventory.

Comment thread dokimi/dokimi/repos/workspace.py Outdated
hop, seen = _canon(link), set()
while hop.is_symlink() and hop not in seen:
seen.add(hop)
hop = _canon(hop.parent / os.readlink(hop))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ee38237. _link_hits walks each path component, so a base link through a removed directory link is caught before _canon resolves it away. The arm refuses that base instead of leaving a dangling AGENTS.md. Test: test_overlay_arm_refuses_a_base_link_through_a_removed_directory_link.

@exai-bot

exai-bot commented Nov 1, 2026 •

Copy link
Copy Markdown

🔍 OpenCodeReview found 3 issue(s) in this PR.

By severity: 2 medium, 1 low

Severity Category Location Finding
medium bug reach.py:131 Must-fix: foreign-only alias groups still produce unsupported per-path findings.
medium bug reach.py:490-491 Must-fix: this reports a negative loading verdict even when the probe has no usable evidence.
low documentation repo-evaluation-contract.md:1063-1064 Nit: document the alias-ambiguity exception here.
Review details
  • Commit reviewed: 3d7334b against merge base b4a0a8c
  • Coverage: 18 files in 4 groups
  • Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
  • Usage: 769k tokens (498k from cache), 53s
  • Tool calls: 59 (47 file_read, 8 code_search, 3 code_comment, 1 file_read_diff)
  • Overlay execution, workspace integration, and tests: 6 files
  • Harness profile overlay configuration and schema: 7 files
  • Guidance reach evaluation and schema: 2 files
  • Overlay documentation and changelog: 3 files

Comment on lines +136 to +140
for n in names.get(r["file"], ()):
if n in foreign:
cross.setdefault(n, {})[r["scenario"]] = r["class"]
if n in own_files:
own_loaded.add(n)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix: a target's tokens do not establish which alias was read. With AGENTS.codex.md -> AGENTS.md and a harness that loads only AGENTS.md, resolve_links/build_workspace seed one shared token pair; this loop nevertheless reports AGENTS.codex.md as cross_loaded. An AGENTS.claude.md alias similarly falsely populates own_loaded. These are incorrect verdicts about overlay support/isolation, not merely alternative labels for the same content. Preserve ambiguous attribution rather than emitting a definitive finding, or collect path-specific access evidence; add a regression where base guidance shares its target with both own and foreign overlay aliases.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a52e675, with a follow-up in 4f33efc. A load of a file named by both a foreign and an own overlay is no longer attributed to either; it is listed under ambiguous. 4f33efc also covers a foreign target with an own alias, which was still counted as an own load. Tests: test_overlay_report_leaves_a_load_of_a_shared_target_unattributed, test_overlay_report_leaves_a_load_of_a_foreign_target_with_an_own_alias_unattributed.

if arm.base == "guidance:bridge" and not bridge:
name = profile.name if profile is not None else "no harness"
raise PrepError(f"guidance:bridge needs a harness profile with a `bridge` key; {name} has none")
overlay = profile.overlay if profile is not None else None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix: the workspace treatment now depends on profile.overlay, but batch_ledger.execution_id omits it and runner.run_cell reuses a completed result before calling prepare_workspace. Run guidance:overlay with a custom profile selecting claude, then change only overlay to codex and resume the same batch: the execution id and cell name are unchanged, so the claude result is returned as skipped instead of running the codex treatment. Validate/pin the overlay selection for overlay cells before reuse (or incorporate it into their identity), with a resume regression. This can be scoped to the new arm to preserve existing non-overlay cell identities.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f442d5d. An overlay cell's execution id now includes the overlay it keeps, so a completed result for another treatment is not reused. Plain-arm reruns still match in a batch holding overlay cells (both arm ids are checked). Test: test_an_overlay_cell_is_identified_by_the_overlay_it_keeps.

Comment thread dokimi/dokimi/repos/workspace.py Outdated
Comment on lines +463 to +464
if arm.base == OVERLAY_ARM:
return _apply_overlay(ws, tracked, overlay or "")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix: tracked comes from the snapshot's index, but setup patches have already been applied with git apply (without updating that index). With a base containing AGENTS.claude.md and a setup patch adding AGENTS.codex.md, the new foreign overlay is absent from tracked, survives this arm, and is committed by the subsequent git add -A. A setup patch deleting an existing foreign overlay instead leaves a stale entry that causes unlink() to raise FileNotFoundError. Enumerate the post-setup workspace files for this arm, or refresh the index before enumeration, and cover both additions and deletions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ee38237. The overlay check reads the workspace after setup patches, not the snapshot index, so an overlay a patch creates is inventoried. Test: test_overlay_arm_reads_the_workspace_after_setup_patches_not_the_index.

Comment thread dokimi/dokimi/repos/workspace.py Outdated
Comment on lines +544 to +546
hop = _canon(hop.parent / os.readlink(hop))
if hop in gone:
return True

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix: resolving the parent here erases removed overlays used as directory links. For example, with AGENTS.codex.md -> docs/, AGENTS.md -> AGENTS.codex.md/base.md, docs/base.md a regular file, and a regular AGENTS.claude.md, this walk jumps directly to docs/base.md and never matches gone. The arm then removes AGENTS.codex.md and leaves the kept AGENTS.md dangling, despite its link guard. Check symlink components before canonicalizing them (or reject foreign overlays that name directories), and add a regression for this case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ee38237. The parent is no longer resolved through a removed overlay used as a directory link; the walk expands symlinks itself and stops on loops (_MAX_HOPS). Tests: test_overlay_arm_refuses_a_base_link_through_a_directory_link, test_overlay_arm_does_not_hang_on_a_symlink_loop_in_the_base.

Comment on lines +552 to +554
kept = sorted(p for p in tracked if overlay_of(p) == own)
if not kept:
raise PrepError(f"guidance:overlay: the base has no AGENTS.{own}.md to keep")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix: a tracked own-overlay name is accepted even if it cannot supply a file. For example, keep a regular AGENTS.md and track AGENTS.claude.md -> missing.md: kept is nonempty, _link_hits finds no removed overlay, and preparation succeeds with only usable base guidance. A cyclic link or a link to a directory also passes. _checkout_tree preserves these links and no earlier preparation step validates their targets. Validate that kept overlays resolve to usable files inside the workspace and reject unusable targets with PrepError; add dangling/cyclic/directory own-overlay regressions (the current cycle test covers unrelated links only).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ee38237. A kept own overlay must now resolve to a regular file in the workspace, otherwise the arm raises PrepError ("does not resolve to a usable file in the workspace"). Test: test_overlay_arm_refuses_an_own_overlay_that_supplies_no_file.

@codecov-commenter

codecov-commenter commented Nov 1, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 93.33333% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.04%. Comparing base (b4a0a8c) to head (3d7334b).

Files with missing lines Patch % Lines
dokimi/dokimi/repos/workspace.py 85.24% 5 Missing and 4 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #338      +/-   ##
==========================================
- Coverage   94.05%   94.04%   -0.01%     
==========================================
  Files         297      298       +1     
  Lines       38018    38145     +127     
  Branches     6424     6453      +29     
==========================================
+ Hits        35756    35872     +116     
- Misses       1452     1459       +7     
- Partials      810      814       +4     
Files with missing lines Coverage Δ
dokimi/dokimi/commands/repo_run.py 94.38% <ø> (ø)
dokimi/dokimi/repos/batch_ledger.py 95.34% <100.00%> (+0.07%) ⬆️
dokimi/dokimi/repos/harness_profile.py 96.36% <100.00%> (+0.13%) ⬆️
dokimi/dokimi/repos/overlays.py 100.00% <100.00%> (ø)
dokimi/dokimi/repos/reach.py 97.08% <100.00%> (+0.47%) ⬆️
dokimi/dokimi/repos/runner.py 96.86% <100.00%> (+0.01%) ⬆️
dokimi/dokimi/repos/workspace.py 89.55% <85.24%> (-0.62%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The arm walks each path component before resolving it, so a removed directory
link in the middle of a base link is a hit; an own overlay must resolve to a file
inside the workspace; the arm lists the workspace (index and untracked files), so
a setup patch applied with git apply is seen.

Thread: PRRT_kwDOVBg7g86yWjv2
Thread: PRRT_kwDOVBg7g86yWjwG
Thread: PRRT_kwDOVBg7g86yWj4D
Thread: PRRT_kwDOVBg7g86yWj4E
Thread: PRRT_kwDOVBg7g86yWj4G
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…y report

A harness that loads a file of a native name, or a file that an own and a foreign
overlay both name, cannot be pinned on one alias: those aliases are listed as
ambiguous, not as own or cross loads.

Thread: PRRT_kwDOVBg7g86yWj4A
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A resumed batch reused an overlay cell that kept another agent's overlay, since
the execution id did not carry the overlay. The id now includes the profile's
overlay for cells of the overlay arm only; other cells keep their ids.

Thread: PRRT_kwDOVBg7g86yWj4C
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The overlay report's ambiguous key was not in the guidance_reach schema.

Thread: PRRT_kwDOVBg7g86yWj4A
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A foreign target with an own alias was counted as an own load and a
cross load of the target. The ambiguity check now looks at the target
too, and an ambiguous link pins no name, as the docstring says.

Thread: PRRT_kwDOVBg7g86yWj4A
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment thread dokimi/dokimi/repos/reach.py Outdated
ambiguous: set[str] = set()
for target, aliases in links.items():
kinds = {"own" if overlay_of(n) == own else "foreign" for n in (target, *aliases) if overlay_of(n) is not None}
if aliases and (PurePosixPath(target).name in native or {"own", "foreign"} <= kinds):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix: native-name ambiguity is checked only on the target. With CLAUDE.md -> docs/shared.md and AGENTS.codex.md -> docs/shared.md, resolve_links supplies both names as aliases. A Claude harness loading only CLAUDE.md produces the shared tokens, but this branch attributes them to AGENTS.codex.md and reports a false cross-load. Check native names across (target, *aliases) and add this shared-native-alias regression; the existing native-target test does not cover it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3d7334b: a group of link names pins nothing when the target or an alias has a harness-native file name (_pinned_names), so a Claude load of docs/shared.md is not attributed to AGENTS.codex.md.
Test: test_overlay_report_does_not_pin_a_shared_target_read_through_a_native_alias (red on 4f33efc).

Comment thread dokimi/dokimi/repos/workspace.py Outdated
hops += 1
if hops > _MAX_HOPS:
return False
target = PurePosixPath(os.readlink(cand))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix: os.readlink returns filesystem-native paths, but PurePosixPath does not split Windows backslashes. On Windows, with AGENTS.codex.md -> docs and a kept AGENTS.md -> AGENTS.codex.md\base.md, the entire target becomes one component. cand then names the regular base.md through the directory link, so the walker never checks the removed AGENTS.codex.md component. Preparation removes that link and leaves AGENTS.md dangling. Parse readlink results with native Path semantics and add a Windows regression for a backslash-separated target through a removed directory link.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3d7334b: readlink targets are read with native Path semantics, so a Windows target such as AGENTS.codex.md\base.md splits into its names.
Test (Windows only, skipped here): test_overlay_arm_refuses_a_backslash_base_link_through_a_removed_directory_link. Not run on Windows in this round.

…rgets use native separators

A shared target reached through a native alias (CLAUDE.md) is not pinned to the foreign alias.
A Windows link target with backslashes splits into names when read as a native path.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment on lines +1063 to +1064
line. `own_loaded` lists the harness's own overlays that loaded in some scenario;
when none did, the text report says so. `own` is the profile's `overlay`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

documentation · low
Nit: document the alias-ambiguity exception here. With AGENTS.claude.md -> AGENTS.md and a Claude profile, reported tokens from AGENTS.md leave own_loaded empty but populate ambiguous; format_matrix deliberately suppresses the 'no overlay loaded' note. Thus an empty own_loaded does not always mean the own overlay was not loaded, and the text report does not always say so. Describe ambiguous and qualify both the own-load and cross-load claims so readers can distinguish an unobserved load from an unattributable one.

for target, aliases in links.items():
kinds = {"own" if overlay_of(n) == own else "foreign" for n in (target, *aliases) if overlay_of(n) is not None}
# a native name anywhere in the group (the target or an alias) cannot tell which file a harness read
if aliases and (any(PurePosixPath(n).name in native for n in (target, *aliases)) or {"own", "foreign"} <= kinds):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix: foreign-only alias groups still produce unsupported per-path findings. For example, overlay_report([{"file": "docs/shared.md", "scenario": "cwd=.", "class": "loaded"}], "claude", {"docs/shared.md": ("AGENTS.codex.md", "AGENTS.copilot.md")}, native={"CLAUDE.md", "AGENTS.md"}) reports both aliases in cross_loaded. A harness reading only one alias produces exactly this shared-token record, so the other finding: is false. The same issue occurs with a foreign overlay target and another foreign overlay alias. Treat these competing overlay names as ambiguous (or report a group-level finding), and add regressions for both arrangements rather than attributing the load to every name.

Comment on lines +490 to +491
if overlays.get("own_files") and not overlays.get("own_loaded") and not overlays.get("ambiguous"):
lines.append(f"note: no {overlays['own']} overlay ({', '.join(overlays['own_files'])}) was loaded in any scenario")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix: this reports a negative loading verdict even when the probe has no usable evidence. For example, with a regular AGENTS.claude.md and every scenario timing out, classify() produces only error records, overlay_report() leaves own_loaded empty, and this prints “no claude overlay ... was loaded in any scenario.” Error records explicitly mean loading is unknown, not not_loaded. Gate this note on usable negative records for the own overlays (and qualify incomplete runs), or report that loading could not be determined. Add a regression for an all-error run.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 3d7334b. The arm and the link walk hold up under the cases I tried, and the earlier bot findings are fixed. Five points are worth fixing before merge: one wrong note in the reach text report, three gaps in tests that pin a reach verdict or the dangling-link refusal, and one mismatch between the contract and the arm.

What I ran, on a Linux review host at 3d7334b:

  • make ci-local: 28 passed, 2 failed, 7 skipped. The two failures are the gitleaks download (Install gitleaks, HTTP 504) and the Scan commits step that needs it. Neither touches this PR. Changed-line coverage passes.
  • tests/test_repo_overlay.py: 40 passed, 1 skipped (the Windows-only test). The tests write only under tmp_path.
  • Reproductions for the reach findings, with the LLM transport mocked.
  • Mutation runs against the new code: 13 mutants of reach.py, 2 of which pass the three reach/overlay test files; a set of workspace.py link-walk mutants, 6 of which pass test_repo_overlay.py.

Findings not anchored inline:

  1. The arm does not require a base AGENTS.md, but the contract says it keeps one. docs/repo-evaluation-contract.md (the overlay bullet, around line 318) says the workspace "keeps the base AGENTS.md" and that an overlay sits "beside an AGENTS.md". _apply_overlay only requires that an own overlay exists anywhere. A repo holding only AGENTS.claude.md and AGENTS.codex.md runs: the call returns (['AGENTS.codex.md'], {'arm_overlay': {'name': 'claude', 'kept': ['AGENTS.claude.md']}}). A repo with AGENTS.md, sub/AGENTS.claude.md and other/AGENTS.codex.md keeps the sub overlay although other/ has no base. The cell is then an overlay alone, which the contract sentence "so a cell cannot be the base alone by accident" does not cover. Suggestion: refuse when no AGENTS.md exists, or reword the contract and say an overlay alone is allowed. The PR description lists this as open question 2, so it needs a decision before merge.
Bot points I checked that no longer apply at this head (10)
  • Copilot's point on setup-patch additions and deletions missed by the index inventory does not apply: _workspace_files lists the workspace after the patches (git ls-files --cached --others, existing paths only), and test_overlay_arm_reads_the_workspace_after_setup_patches_not_the_index pins it.
  • Copilot's point on a base link through a removed directory link does not apply: _link_hits walks each path component before resolving, and the arm refuses AGENTS.md -> AGENTS.codex.md/base.md instead of leaving it dangling.
  • exai-bot's point on attributing a shared target to the wrong alias does not apply: a load of a file named by an own and a foreign overlay is listed under ambiguous, not as an own or cross load.
  • exai-bot's point on resuming a batch after changing only overlay does not apply: the overlay arm's execution id now carries the overlay, and plain cells keep their ids.
  • exai-bot's point on the index being stale after setup patches does not apply: the inventory comes from the workspace, as above.
  • exai-bot's point on the parent walk erasing a removed directory link does not apply: the walk expands symlinks itself and stops on loops.
  • exai-bot's point on an unusable own overlay does not apply: a kept overlay that is missing, cyclic or a directory link raises PrepError before anything is removed.
  • exai-bot's point on a shared native alias such as CLAUDE.md does not apply: _pinned_names pins nothing when the target or any alias has a native file name.
  • exai-bot's point on backslash readlink targets does not apply: targets are read with native Path semantics. The Windows test is skipped on Linux, so I could not run it.
  • exai-bot's point on foreign-only alias groups does not apply: with docs/shared.md named by AGENTS.codex.md and AGENTS.copilot.md, both are listed in cross_loaded, but a foreign overlay was read in either case, so the finding that isolation failed is right. Which alias was read is the documented alias attribution the description declines to narrow.

where = ", ".join(f"{sid} ({cls})" for sid, cls in seen.items())
lines.append(f"finding: {path}, an overlay for another agent, was loaded by {payload['harness']['name']} in {where}")
if overlays.get("own_files") and not overlays.get("own_loaded") and not overlays.get("ambiguous"):
lines.append(f"note: no {overlays['own']} overlay ({', '.join(overlays['own_files'])}) was loaded in any scenario")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The text report says no own overlay was loaded when the probe had no usable evidence. classify() gives error for a scenario that could not be read, and the contract treats error as "loading unknown", not not_loaded. With a regular AGENTS.claude.md and one scenario that errored, format_matrix prints:

AGENTS.claude.md   E
note: no claude overlay (AGENTS.claude.md) was loaded in any scenario
error in cwd=.: timeout

(overlay_report([{"file": "AGENTS.claude.md", "scenario": "cwd=.", "class": "error"}], "claude", {"AGENTS.claude.md": ()}, ...) gives own_loaded: [].) A reader takes that as a negative measurement of the harness.

Suggestion: print the note only when at least one scenario for an own overlay file was not_loaded and none was error; otherwise print note: could not determine whether the claude overlay loaded. Add a regression for an all-error run.

for path, seen in (overlays.get("cross_loaded") or {}).items():
where = ", ".join(f"{sid} ({cls})" for sid, cls in seen.items())
lines.append(f"finding: {path}, an overlay for another agent, was loaded by {payload['harness']['name']} in {where}")
if overlays.get("own_files") and not overlays.get("own_loaded") and not overlays.get("ambiguous"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ambiguous exemption in this condition is not pinned by any test. Mutation check: removing and not overlays.get("ambiguous") leaves test_repo_overlay.py, test_repo_reach.py and test_repo_reach_edges.py at 148 passed, 1 skipped. test_text_report_states_the_cross_load_as_a_finding uses a payload with no ambiguous key. With the exemption gone, a run where AGENTS.claude.md -> AGENTS.md loaded through the shared file prints "no claude overlay ... was loaded", a false negative on the check.

Suggestion: add a format_matrix test with own_files=["AGENTS.claude.md"], own_loaded=[], ambiguous=["AGENTS.claude.md"] that asserts the note is absent.

"unresolved": unresolved,
"scenarios": scenario_rows,
"records": records,
"overlays": overlay_report(records, profile.overlay, {s.path: s.aliases for s in seeded},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The run_reach wiring of overlay_report has no test. No test in tests/test_repo_reach*.py or test_repo_overlay.py calls run_reach and reads payload["overlays"]. Mutation check: replacing {PurePosixPath(n).name for n in profile.guidance_names} with set() still gives 148 passed, 1 skipped. With native empty, a CLAUDE.md -> AGENTS.codex.md alias is pinned as a cross load, and the report prints a false finding: line.

Suggestion: add a test that runs run_reach with a mocked transport on a repo holding AGENTS.claude.md and AGENTS.codex.md, and asserts payload["overlays"]["own"], cross_loaded and own_loaded. Validate that payload against guidance_reach.v1 too.

raise PrepError(f"guidance:overlay: the base has no AGENTS.{own}.md to keep")
for rel in kept: # the arm keeps it, so it must supply a file: a dangling, cyclic or directory link does not
real = Path(os.path.realpath(ws / rel))
if not real.is_file() or not real.is_relative_to(ws.resolve()):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dangling-link refusal and the kept-overlay checks are only partly pinned by tests. Each of these mutants of _canon, _link_hits and _apply_overlay passes all of test_repo_overlay.py (40 passed, 1 skipped):

  • drop or not real.is_relative_to(ws.resolve()) here, so a kept overlay can link outside the workspace;
  • drop and not is_symlink() in the directory check;
  • _canon returns path unresolved;
  • the .. branch no longer steps up;
  • the absolute-target branch no longer resets to the anchor;
  • _link_hits starts at link.parent instead of its real path.

The current code refuses each of these correctly when I run it (an absolute AGENTS.md -> /abs/ws/AGENTS.codex.md, ../ws/AGENTS.codex.md, ./sub/../AGENTS.codex.md, a two-link chain). So this is a test gap, but the lines decide whether the arm refuses a base that would dangle, and the only backslash test is skipped on Linux.

Suggestion: add parametrized refusal tests for an absolute target, a .. target, a ./x/.. target, a two-link chain, and an own overlay linking outside the workspace.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants