Skip to content

feat(repo): record public_at and guidance:rev history for leak checks - #359

Open
antonwolfy wants to merge 9 commits into
intel:mainfrom
antonwolfy:feat/replay-leak-record
Open

antonwolfy wants to merge 9 commits into
intel:mainfrom
antonwolfy:feat/replay-leak-record

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

First half of #290, "repo compare: flag replay tasks a guidance revision may have seen" (issue, decisions). A guidance:rev:<sha> arm runs a later revision of a repository's guidance against review_replay tasks built from earlier PRs. If that guidance was edited after a task's review, the edit can encode the review. The batch has no way to show this today. On main, a replay task.json holds source_pr, source_url, reviewed_commit, merge_commit and left_out_files, and no time. A guidance:rev: cell's meta.json arm_rev is exactly {"rev", "sha", "removed"}, which tests/test_repo_arms.py asserted on main. This PR records both facts. The flags in repo compare are the second PR, as the issue decided; repo compare does not change here.

The second commit settles this PR's first review: four points from five comments. The third commit settles two more comments. f6a9cb9 settles four more, and 3c1f93d, 06f64a6, bcb96c1 and 3f2895a settle the latest four, one commit each. At the commit before each fix:

Input in the history of <rev> Before Now
Any commit, guidance or not, whose subject or path holds \x1e (e.g. subject guidance\x1eedit) first commit: ValueError: not enough values to unpack from workspace preparation; by the code, not run: it is not one of the errors repo run records for a cell (repos/runner.py), so it ends the run Read with NUL framing; subject and paths kept as they are
Cells of one arm that repo run's thread pool prepares together first commit: each cell runs the full-history git log one read per key; the other cells wait for it
A merge whose conflict resolution edits AGENTS.md first commit: not listed; AGENTS.md's guidance_latest is an older parent commit listed with its committer date, and AGENTS.md's guidance_latest
AGENTS.md -> docs/GUIDE.md, a later commit edits only docs/GUIDE.md first commit: not listed; AGENTS.md's guidance_latest is the commit that made the link listed, and AGENTS.md's guidance_latest
Repository-local log.showSignature=true and a signed commit anywhere in the history second commit: PrepError: git log: a commit starts with 'No signature\n<sha>', not a commit id; the cell is not prepared read; the signed commit is listed
Repository-local log.showRoot=false, i18n.logOutputEncoding=UTF-16 or diff.ignoreSubmodules=all second commit: the root commit's files are missing; a PrepError on the UTF-16 output; a commit that adds a gitlink at a guidance path is missing the same arm_rev as without the key
AGENTS.md -> docs/current -> old.md, a later commit only retargets docs/current to an existing review-derived.md second commit: not listed; AGENTS.md's guidance_latest is the commit that added review-derived.md listed, and AGENTS.md's guidance_latest
Two gold threads, one at 2026-10-02T12:00:00Z and one at 2026-10-01T00:00:00 (no offset) f6a9cb9: public_at is 2026-10-02T12:00:00Z, though the thread without an offset may have come first no public_at; the build notes no public_at: no first-comment createdAt with an offset on pr7-t2
With **/AGENTS.md, a commit that only edits .agent-eval/tasks/x/AGENTS.md (a task fixture) f6a9cb9: listed in guidance_commits, though the arm installs nothing under eval_dir not listed

Changes

  • repo tasks replay writes public_at into each task.json (repos/replay/build.py::_public_at). It is the earliest createdAt of a gold thread's first comment, kept as the threads file gives it. Times are compared as datetimes, not strings: 2026-10-02T20:00:00+09:00 is earlier than 2026-10-02T12:00:00Z. A thread the task leaves out does not count. GitHub's Z is UTC and counts. When any gold thread's time is missing, not ISO 8601 or has no offset, the earliest time is unknown: the key is absent, and the build notes name those threads.

  • New repos/guidance_history.py: for a guidance:rev:<rev> arm, meta.json arm_rev now also holds:

    • guidance_latest: for each guidance file the arm wrote, the commit reachable from <rev> that last touched it, or for a link it or any path its resolution goes through (each intermediate link and the file it resolves to), by committer date. Not capped.
    • guidance_commits: the commits reachable from <rev> that touch a path the guidance globs match outside eval_dir, or a path a guidance link resolves through, newest first by committer date, each with its subject. At most GUIDANCE_COMMITS_CAP (200).
    • guidance_commits_truncated: true when the cap cut the list.

    Each commit carries sha, authored (%aI) and committed (%cI), a UTC date written Z whichever form the git that read it prints (git 2.43 prints +00:00, 2.51 Z). The history comes from one git log -z --raw --no-abbrev --no-renames --diff-merges=dense-combined --no-show-signature --root --encoding=UTF-8 --ignore-submodules=none over <rev>, filtered with match_glob. It is memoized per (git dir, revision, globs, eval_dir, links), so every cell of an arm reads it once. The output is split on NUL as bytes and each field decoded on its own, so a \r in a subject or path is kept. A shallow git dir is refused with a PrepError naming git fetch --unshallow: its history stops at the boundary commit and would name a later commit as the last to touch a guidance file.

  • Review of this PR (second commit):

    • Framing (comment, comment). _parse_log reads the -z --raw output. Every field ends with a NUL. The four header fields are taken by position, and each file is a :-led status record followed by exactly one path. A path that looks like a commit id or a status record (:M) stays a path. Git cuts a subject at a NUL (git commit refuses one), and a header field that is not a commit id is a PrepError, not a ValueError.
    • One read under parallel cells (comment). _guidance_commits holds a lock per key around the lru_cached read, so one thread reads and the others take its result. Different keys still read in parallel.
    • Merges (comment). With --diff-merges=dense-combined, a merge counts for a path its result changes against every parent: a conflict resolution, or an edit made in the merge. A merge that takes a file as one parent has it is not listed; the commits it merged are. With git 2.51, --raw also lists a clean merge that joins its parents' edits to different parts of one file, because the result differs from each parent. A test pins this, and the docstring and contract say so.
    • Link targets (comment). _apply_rev passes _guidance_at's links to arm_rev_history. A commit that touches a link's target counts for the link's path, and the links are part of the cache key.
  • Review of this PR (third commit):

    • The repository's own config (comment). _git_log turns off the system and global config, but a local source's own git dir keeps its config. With git 2.51, four of the keys a repository can set change this command's output, and each is now overridden by a flag (guidance_history._PINNED): log.showSignature (--no-show-signature), log.showRoot=false (--root), i18n.logOutputEncoding (--encoding=UTF-8) and diff.ignoreSubmodules=all (--ignore-submodules=none). Set one at a time, about 40 other log.*, diff.*, color.*, core.* and format.* keys left the output byte for byte the same. Among them are color.ui=always, diff.relative, log.decorate, log.follow and core.abbrev. The keys that the explicit --no-renames, --diff-merges and --format already override (diff.renames, log.diffMerges, format.pretty) need nothing.
    • Every link of a chain (comment). _guidance_at now also returns, for each guidance link, every path its resolution goes through: the intermediate links, then the final file. The walk moved into a new _link_chain, which lowers _guidance_at's complexity from 11 to 5 and removes its entry from scripts/growth_baseline.json. _apply_rev passes those chains to arm_rev_history. A commit that touches any path of a chain counts for the link's path, and the chains are part of the cache key. _apply_rev still installs the final file's bytes.
  • Review of this PR (3c1f93d, 06f64a6, bcb96c1, 3f2895a):

    • public_at from every gold thread (comment, 3c1f93d). _public_at returns None when any gold thread's first-comment createdAt is missing, not ISO 8601 or has no offset, and appends a build note naming those threads. The minimum of the others could be later than the real time, and a guidance commit between the two would then read as made before the answer was public. Contract §4.6, the docstring and the changelog fragment say when the key is absent.
    • eval_dir left out (comment, 06f64a6). arm_rev_history takes eval_dir (manifest.run_paths()[0], as _guidance_at uses it), and it is part of the cache and lock key. _counts_for skips a path within it before the glob match. A link chain needs no filter: _link_chain refuses one that reaches into eval_dir. Contract §3 and the changelog fragment say so.
    • Cost of the read (comment, bcb96c1). The log is not narrowed with pathspecs: git log -- <pathspec> turns on history simplification, so it would need --full-history and the dense-combined merge behaviour proved again. Git's :(glob) is also not match_glob, so a pre-filter needs a proof that it keeps every commit the matcher keeps. The Python side was mostly match_glob, which builds its pattern on every call. It now runs once per distinct path of a read. The module docstring states the cost.
    • One git environment, one shallow check (comment, 3f2895a). source.isolated_git_env() is the one policy: PATH, LANG, TMPDIR and SYSTEMROOT kept, no system or global config, HOME at devnull, no prompt. workspace._git_env adds the fixed author and date to it, and the history's reads use it. source.is_shallow(git_dir, env) is the one check, used by base_opener and by the history's refusal. The history passes its own environment, because source's default inherits the caller's, and a GIT_DIR there wins over -C (git 2.51: a shallow clone reads false with GIT_DIR set to a full one). base_opener keeps the caller's environment. No behaviour change.
  • repos/workspace.py::_apply_rev: one call added where arm_rev is built, with no new branch. guidance:none|raw|bridge and patch: are unchanged.

  • Contract: §3 describes the three arm_rev keys, link chains, eval_dir and which merges count. §4.6 adds public_at to the list of fields the replay task.json adds for the reader, and says when it is absent.

  • Changelog fragment (Added).

Risk

The output of repo tasks replay and guidance:rev: cells gains keys; nothing is taken away. A replay task with a gold thread that has no usable time gets no public_at, rather than the earliest of the other threads' times. One input is now refused: a guidance:rev: cell whose local source is a shallow clone fails its preparation (dokimi's own clones of a remote are bare with no --depth, so they are never shallow). The new keys do not change a cell's identity. The cell name comes from task, arm, model and rep. The execution id comes from the agent options. The batch's reuse and replace checks (batch_ledger.reused_result, check_batch_executions) read result.json. None of them reads meta.json arm_rev. So rerunning an existing batch neither refuses nor reruns its finished cells; only cells that run again get the new keys. A guidance:rev: cell now runs one more git log over the revision's whole history, once per process. Its --raw output carries two object ids per file, so it is two to three times the --name-only output, and --diff-merges=dense-combined also diffs every merge against its parents. Measured on bare clones, --name-only against the second commit's form: this repository (485 commits, 5 merges) 0.4 MB against 1.2 MB, both under 0.1 s; dpnp (3,127 commits, 185 merges) 1.0 MB against 2.7 MB, both 0.1 s; numpy (42,470 commits, 10,335 merges) 9.3 MB in 0.67 s against 19.3 MB in 0.88 s. The first commit's text also cited 3.7 s (51 MB) for --name-only over a 9,384-commit repository; the --raw form was not run there. The third commit's four flags only bring back git's default output, so these sizes still hold for a repository that sets none of those keys; they were not measured again. The Python parse of the log is measured on a generated history of 42,000 commits with 5-6 paths each: git log 1.0 s (28.8 MB); parse and match 0.92 s before bcb96c1 and 0.12 s after, with the same 210 commits listed. It still grows with the history, and the other cells of the arm wait for it. The time limit is 1800 s, the same as workspace preparation's other git calls. A merge that cleanly joins two edits to one guidance file is listed although it adds no text of its own, so it can make a file's guidance_latest later than the last commit that wrote text. The same goes for a link's chain: a commit that touches an intermediate link or the final file counts for the link, even when the bytes the arm installs did not change.

Testing

On a Linux review host, at 3f2895a (on main 41b3d8d):

  • make ci-local: 30 passed, 0 failed, 7 skipped. Skipped: external-catalog-smoke (needs docker), test-latest, docs deploy and pip-audit report (job conditions false on a PR), cli-surface "Package the result", conda-forge-deps (no micromamba), semgrep secrets. 12570 tests passed, 5 skipped, 6 xfailed; changed-line coverage 98% (151 lines; the two missed are the git log timeout and failure raises). Install gitleaks and Scan commits ran: no leaks found.

  • At 1b75473 (on main 93a5062): 30 passed, 0 failed, 7 skipped. Skipped: external-catalog-smoke (needs docker), test-latest, docs deploy and pip-audit report (job conditions false on a PR), cli-surface "Package the result", conda-forge-deps (no micromamba), semgrep secrets. 12022 tests passed, 59 skipped, 6 xfailed; changed-line coverage 98% (121 lines; the two missed are the git log timeout and failure raises).

  • At the second commit, d35b581: 30 passed, 0 failed, 7 skipped; 12017 tests passed, 59 skipped, 6 xfailed; changed-line coverage 97% (96 lines). At the first commit, c21791a: 28 passed, 2 failed, 7 skipped. Both failures were the host: Install gitleaks got HTTP 504 four times, so Scan commits had no binary to run (exit 127). The same scan, run separately with CI's pinned gitleaks 8.30.1 (checksum checked) over the commit: no leaks found. Both steps pass at d35b581.

  • New tests: tests/test_repo_guidance_history.py builds a toy history with per-commit GIT_AUTHOR_DATE / GIT_COMMITTER_DATE. It includes a commit authored two months before it was committed, and a commit whose committer date is older than its parent's. The tests cover:

    • the latest commit per file, with both dates;
    • committer date, not author date, choosing a file's latest commit;
    • newest first by committer date rather than git's walk order;
    • no code-only commit in the list;
    • the cap at 2 (2 entries, truncated) and at 4 (all 4, not truncated);
    • an older revision recording only its own history;
    • no history keys on guidance:raw and patch: cells;
    • one git log for two cells of the arm;
    • (second commit) one git log for four threads asking for one key at once;
    • (second commit) subjects holding \x1e and \x1f, a guidance path holding \x1e, a file named like a commit id and a file named :M;
    • (second commit) a log field that should start a commit but is no commit id, which is a PrepError;
    • (second commit) a merge whose conflict resolution edits AGENTS.md, a plain merge of a branch with a guidance commit, and a clean merge of edits to different parts of AGENTS.md;
    • (second commit) an edit to only the target of AGENTS.md -> docs/GUIDE.md;
    • (third commit) a guidance commit signed with an ssh key made by ssh-keygen in the test's temporary directory (gpg.format=ssh), under repository-local log.showSignature=true: it is read, listed and AGENTS.md's guidance_latest. The test skips only without ssh-keygen; it ran here;
    • (third commit) repository-local log.showRoot=false, i18n.logOutputEncoding=UTF-16 and diff.ignoreSubmodules=all, one at a time, over a history with a root commit that adds AGENTS.md, a non-ASCII subject and a commit that adds a gitlink at vendor/AGENTS.md: each gives the same arm_rev as without the key;
    • (third commit) AGENTS.md -> docs/current -> old.md, then a commit that adds docs/review-derived.md, then one that only retargets docs/current to it: the retargeting commit is AGENTS.md's guidance_latest and first in guidance_commits, and the workspace's AGENTS.md holds review-derived.md's text;
    • (06f64a6) with **/AGENTS.md, a commit that only edits .agent-eval/tasks/x/AGENTS.md: not in guidance_commits or guidance_latest, while the root AGENTS.md commit after it is;
    • (3f2895a) a directory git cannot read: still a PrepError (git rev-parse failed) out of the shared shallow check, not source's SourceError.

    In tests/test_repo_replay.py, the tests cover public_at as the earliest gold thread's time, a left-out thread with an earlier time not counting, a Z and +09:00 pair compared as instants, and (3c1f93d) no key, with the note naming the threads, when a gold thread's time has no offset, when one has no createdAt, and when none has a time.

  • Mutants: 15 breaks in a copy of the tree, 17 test runs, and every run failed. The breaks:

    • counting every thread for public_at;
    • comparing the times as strings;
    • writing public_at: null;
    • no sort;
    • sorting by author date;
    • no glob filter;
    • the oldest commit as a file's latest;
    • authored holding the committer date;
    • no cap;
    • >= for truncated;
    • truncated never set;
    • no memoization;
    • git log --all instead of <rev>;
    • the keys dropped from arm_rev;
    • history keys on guidance:raw.
  • Mutants for the second commit: 6 breaks, 7 test runs, and every run failed on its assertion. The breaks:

    • the first commit's \x1e/\x1f split and --name-only log (ValueError);
    • the per-key lock removed (four reads);
    • --diff-merges=dense-combined dropped (the resolution merge and the clean merge are not listed);
    • --diff-merges=first-parent in its place (the plain merge is listed);
    • {} passed in place of the links;
    • the commit-id check dropped.
  • The second commit's new test file against the first commit's code: 6 of its tests fail and 8 pass. The plain-merge test passes there, because that code listed no merge at all. The concurrency test and the commit-id test fail there on the new links argument and the new _parse_log, so the lock and commit-id mutants above are what pin them.

  • Mutants for the third commit: 5 breaks, 5 test runs, and every run failed. The breaks:

    • --no-show-signature dropped (the signed-commit test: PrepError: git log: a commit starts with 'No signature\n<sha>');
    • --root dropped (the log.showRoot=false case);
    • --encoding=UTF-8 dropped (the UTF-16 case: a PrepError);
    • --ignore-submodules=none dropped (the diff.ignoreSubmodules=all case);
    • only the final file recorded for a link (the chain test: AGENTS.md's guidance_latest is the commit that added review-derived.md).
  • The third commit's test file against the second commit's code: its 5 new tests fail and the other 14 pass.

  • Mutants for 3c1f93d..3f2895a: 4 breaks, 4 test runs, and every run failed:

    • the unusable gold thread skipped again (the old continue): the three no-public_at tests fail, and the all-usable test passes;
    • the _is_within(path, eval_dir) skip dropped: the eval_dir fixture test fails (the fixture commit is listed);
    • the history's shallow refusal call dropped (3f2895a): the shallow test fails (DID NOT RAISE PrepError);
    • source's SourceError let out of the shared shallow check: the new not-a-repository test fails (SourceError: git rev-parse failed: fatal: not a git repository).
  • 3f2895a changes no existing test; bcb96c1 changes no output (the same 210 commits on the generated history, compared as JSON before and after).

  • Not run: the new tests against main's code. This is a new feature, and on main they fail at import (dokimi.repos.guidance_history does not exist).

  • Merged together at f6a9cb9 with the branches for review_replay: optional judged recall over the credited hits #291 (judged recall, 8e05118), novelty: spread the atom cap across the artifact instead of its opening #293 (novelty cap spread, 82d576a) onto main 41b3d8d: the full suite (12641 passed, 5 skipped, 6 xfailed), mypy (318 files) and ruff check pass.

  • make ci-local is green at f6a9cb9 (30 passed, 0 failed).

Revert

Revert the commits. The public_at and arm_rev keys already written into task directories and batches stay. Nothing reads them yet.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

Settles the recording half of intel#290 ("repo compare: flag replay tasks a
guidance revision may have seen"), as decided on the issue: committer
date, not author date; the field is public_at; the per-file latest
commit is uncapped and the newest-first list is capped.

- repo tasks replay writes public_at into each task.json: the earliest
  createdAt of a gold thread's first comment, compared as times, as the
  threads file gives it. Threads the task leaves out do not count; with
  no timed gold thread the key is absent.
- A guidance:rev:<rev> arm's meta.json arm_rev adds guidance_latest (the
  commit reachable from <rev> that last touched each guidance file at
  <rev>), guidance_commits (the commits touching a path the guidance
  globs match, newest first by committer date, at most 200) and
  guidance_commits_truncated, each commit with %aI and %cI dates. One
  git log per (git dir, revision, globs) in a process, in the new
  repos/guidance_history.py.

The keys are records only: meta.json feeds neither the cell name, the
execution id nor the batch's reuse and replace checks, so an existing
batch resumes as before. repo compare is unchanged; its flags come in a
later change.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 1, 2026 19:28
@antonwolfy
antonwolfy requested a review from a team as a code owner November 1, 2026 19:28
@exai-bot

exai-bot commented Nov 1, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 8 selected item(s).

Review details
  • Commit reviewed: 3f2895a against merge base f6a9cb9
  • Coverage: 8 files in 1 groups
  • Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
  • Usage: 95k tokens (45k from cache), 24s
  • Tool calls: 10 (7 file_read, 3 code_search)
  • Repository replay leak tracking and guidance history contract: 8 files

Comment thread dokimi/dokimi/repos/guidance_history.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Valid subjects or filenames containing the record delimiter can abort guidance revision runs, while history collection omits symlink-target and merge-resolution edits and permits duplicate full-history reads under concurrency.

4 open issues
What changed in this PR

4 new findings.

This PR records replay review times and guidance revision history for future leak checks under #290. It adds metadata without changing repo compare.

Changes:

  • Records optional public_at from the earliest valid gold-thread timestamp.
  • Adds per-file latest commits and capped guidance history to guidance:rev: metadata.
  • Documents the new fields and adds timestamp, history, and caching tests.
File Description
dokimi/​tests/​test_repo_replay.py Tests timestamp ordering, excluded threads, and missing times.
dokimi/​tests/​test_repo_guidance_history.py Tests history dates, ordering, caps, revision scope, and caching.
dokimi/​tests/​test_repo_arms.py Allows additional revision metadata while preserving existing assertions.
dokimi/​dokimi/​repos/​workspace.py Attaches guidance history to revision-arm metadata.
dokimi/​dokimi/​repos/​replay/​build.py Computes and writes optional public_at.
dokimi/​dokimi/​repos/​guidance_history.py Collects, filters, sorts, and caches guidance commits.
dokimi/​docs/​repo-evaluation-contract.md Defines the new task and arm metadata fields.
dokimi/​changelog.d/​20261101_120000_antonwolfy_replay-leak-record.md Announces metadata for future leak checks.

Review effort: Balanced

Comment thread dokimi/dokimi/repos/guidance_history.py Resolved
Comment thread dokimi/dokimi/repos/guidance_history.py Resolved
Comment thread dokimi/dokimi/repos/guidance_history.py Resolved
Comment thread dokimi/dokimi/repos/workspace.py Resolved
@antonwolfy antonwolfy self-assigned this Nov 1, 2026
Settles the review of intel#359 on the guidance history a guidance:rev: arm
records in meta.json arm_rev (intel#290, first half).

- The log was split on \x1e and \x1f, which a subject or a path may hold:
  one such commit anywhere in the history raised ValueError and aborted
  the arm's workspace preparation. It is now read with -z --raw, where
  every field ends with a NUL and each file is a ':'-led status record
  followed by its one path, so a path is never read as a header nor a
  header as a path. Git cuts a subject at a NUL; a header that does not
  start with a commit id is a PrepError.
- lru_cache did not coalesce concurrent misses, so cells of one arm that
  repo run's thread pool prepared together each ran the full-history log.
  A lock per key now lets one thread read while the others wait for its
  answer; different keys still read in parallel.
- A merge listed no files, so a guidance edit made while resolving a
  merge was in neither output. With --diff-merges=dense-combined a merge
  counts for a path its result changes against every parent (a conflict
  resolution, an edit made in the merge, or a clean merge of edits to
  different parts of one file); a merge that takes a file from one parent
  still does not.
- The arm installs a guidance link as its target's bytes, but only paths
  the globs match were counted. The links now reach arm_rev_history and
  are in its cache key: a commit touching a link's target counts for the
  link's path.

The contract (section 3) and the changelog fragment say what is counted.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@codecov-commenter

codecov-commenter commented Nov 1, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 97.35099% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.30%. Comparing base (41b3d8d) to head (3f2895a).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
dokimi/dokimi/repos/guidance_history.py 95.74% 2 Missing and 2 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff            @@
##             main     #359    +/-   ##
========================================
  Coverage   94.29%   94.30%            
========================================
  Files         308      309     +1     
  Lines       39322    39441   +119     
  Branches     6651     6667    +16     
========================================
+ Hits        37079    37193   +114     
- Misses       1430     1434     +4     
- Partials      813      814     +1     
Files with missing lines Coverage Δ
dokimi/dokimi/repos/replay/build.py 98.15% <100.00%> (+0.22%) ⬆️
dokimi/dokimi/repos/source.py 95.61% <100.00%> (+0.06%) ⬆️
dokimi/dokimi/repos/workspace.py 90.43% <100.00%> (+0.26%) ⬆️
dokimi/dokimi/repos/guidance_history.py 95.74% <95.74%> (ø)

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread dokimi/dokimi/repos/guidance_history.py Resolved
Comment thread dokimi/dokimi/repos/workspace.py Resolved
Settles two review points on the guidance history of a guidance:rev:
arm (intel#290, recording half).

The repository's own config could change what the history's git log
prints. log.showSignature=true put verification text outside the
NUL-framed fields, so one signed commit anywhere in the history stopped
the arm's workspace from being prepared. Checked against git 2.51, four
local keys change the output: log.showSignature, log.showRoot=false
(the root commit with no files), i18n.logOutputEncoding (a UTF-16
subject holds NULs) and diff.ignoreSubmodules=all (a commit that adds a
gitlink is dropped). The log now passes --no-show-signature, --root,
--encoding=UTF-8 and --ignore-submodules=none.

A guidance link resolved through intermediate links (AGENTS.md ->
docs/current -> old.md) recorded only its final file, so a commit that
only retargets docs/current changed what the arm installs but was in
neither history output. _guidance_at now also returns every path each
link resolves through (split into _link_chain, which lowers its
complexity below the baseline), and a commit that touches any of them
counts for the link's path and is part of the cache key. The arm still
installs the final file's bytes.

Tests with real repositories: a signed commit under repository-local
log.showSignature=true, each other pinned key leaving arm_rev as it is
without it, and a commit that only retargets an intermediate link.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

Review rounds at 1b75473, one commit per round, each thread answered and resolved:

  • d35b581: the guidance history's git log is NUL-framed (-z --raw), so a \x1e or \x1f in any subject or path no longer aborts workspace preparation. Cells of one arm that start together read the history once (a lock per key). A merge whose own result changes guidance counts (--diff-merges=dense-combined), and a commit that touches a guidance link's target counts for the link.
  • 1b75473: four repository-local keys that changed the log's output are pinned by flags (--no-show-signature, --root, --encoding=UTF-8, --ignore-submodules=none), each tested with a real repository. Every path a guidance link resolves through counts, intermediate links included.

The --raw form, measured on bare clones: numpy (42,470 commits, 10,335 merges) gives 19.3 MB in 0.88 s, against 9.3 MB in 0.67 s for --name-only.

make ci-local is green at each commit. With #357 and #358 at their heads, merged together onto main ff646be, the full suite (12203 passed), mypy and ruff pass. The description is updated.

@napetrov napetrov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: comment only, not approved. One blocking thread (a local shallow checkout gives a silently wrong arm history), two nits, one question. Head 1b75473, merge base 93a5062, lane L2. CI: 0 failing Actions jobs.

# Finding Label Provenance Scope Probe
1 local shallow checkout: arm history stops at the boundary commit blocking introduced new surface probes/lead_verify.py (a1)
2 a CR in a guidance path or subject drops the commit nit introduced new surface probes/lead_verify.py (b2)
3 date tests expect Z; git 2.43 prints +00:00 question - - probes/lead_verify.py (b3)
4 _public_at docstring says Z counts as none nit introduced new surface probes/d4.py

Raised before: T1 to T7 (exai-bot, copilot) are fixed on head; no new thread repeats them. Finding 2 is near T1 (framing) but is a different root cause (decoding).

Unverified: a real repo run with a shallow source (precondition of 1); git 2.51 pins and the body's timings (not probed); eval_dir-only commits in guidance_commits (contract reads as intended).

Coverage: diff, contract doc and test assertions read; passes at level high with VERIFY on 3 candidates plus the docstring, probed on head (new code, no base); prior threads checked by reading. Not run: test suite, lenses. Not looked at: full-history git log cost on large repos.

Comment thread dokimi/dokimi/repos/guidance_history.py Resolved
Comment thread dokimi/dokimi/repos/guidance_history.py Outdated
keep = ("PATH", "LANG", "TMPDIR", "SYSTEMROOT")
env = process.Env.only(keep, {"GIT_CONFIG_NOSYSTEM": "1", "GIT_CONFIG_GLOBAL": os.devnull, "HOME": os.devnull,
"GIT_TERMINAL_PROMPT": "0"})
done = process.git("-C", str(git_dir), "log", "-z", "--raw", "--no-abbrev", "--no-renames",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: a CR in a guidance path or subject drops the commit from the history
process.git returns stdout through _decode (dokimi/dokimi/process.py), which turns each \r into \n: the path then matches no glob and the commit disappears without a report.
How to check: commit subject guidance\redit touching a\rb/AGENTS.md; arm_rev_history gives no guidance_latest and no guidance_commits (probes/lead_verify.py b2, head only: new code).
Expected: a\rb/AGENTS.md latest at that commit and the subject kept, as the PR body's framing table promises for the record delimiter.
Fix: read _git_log as bytes and split the fields before decoding; add a test with a CR in a path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f6a9cb9. _git_log now returns stdout_bytes, and _parse_log splits on NUL as bytes, then decodes each field as UTF-8 with replace. That is the same decoding dokimi.process uses, so a path that is not UTF-8 gives the same string as the arm's own file listing. New test test_a_carriage_return_in_a_guidance_path_or_a_subject_is_kept: a commit with subject guidance\redit that touches a\rb/AGENTS.md is that path's guidance_latest, and the subject keeps its \r.

End to end, the arm's own file listing (_guidance_at's ls-tree, unchanged from main) still reads through process's text decoding. So prepare_workspace does not install a guidance file with a \r in its path, and that path is not in guidance_latest. The commit is listed in guidance_commits. That listing is on main and is left for a separate change.

Comment thread dokimi/dokimi/repos/replay/build.py Resolved
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Settles four review points on the guidance history a guidance:rev: arm
records in meta.json arm_rev (intel#290, recording half).

- A local source is used as it is, and in a shallow checkout git log
  takes the boundary commit for a root: with guidance v1 -> v2 -> a code
  commit cloned at --depth 1, the code commit was recorded as the last
  to touch AGENTS.md and the only guidance commit. The history now asks
  git rev-parse --is-shallow-repository first and raises PrepError for a
  shallow git dir, naming it and git fetch --unshallow. dokimi's own
  clones of a remote are bare and never shallow; a blob-less clone keeps
  working, as log --raw compares object ids only.
- process.git's text output turns each \r into \n, so a guidance path or
  subject holding a \r lost it, and the commit matched no glob and was
  dropped without a report. The log is now split on NUL as bytes and
  each field decoded as UTF-8 on its own (with replacement, as
  dokimi.process decodes).
- git 2.51 prints a UTC %aI / %cI as Z, git 2.43 as +00:00. A UTC date is
  now recorded as Z whichever git read it; other offsets stay as printed.
- The _public_at docstring said a Z time counts as none. A time without
  an offset counts as none; GitHub's Z is UTC and counts, now pinned by a
  test.

The contract (section 3) and the changelog fragment name the refusal and
the date form.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

Review round at f6a9cb9, on top of a merge of main at 41b3d8d (fd27a2b, clean):

  • f6a9cb9: a shallow git dir is refused with a PrepError naming git fetch --unshallow, since its history stops at the boundary commit. The log is split on NUL as bytes and each field decoded on its own, so a \r in a subject or path is kept. A UTC date is recorded as Z whichever form git prints (2.43 prints +00:00). The _public_at docstring now says Z counts.

The \r thread stays open: the history keeps the \r, but the arm's own guidance listing on main (_guidance_at's ls-tree) still decodes \r as \n, so such a file is not installed. That listing predates this PR; I'd fix it in a separate change unless you want it here.

CI is green on f6a9cb9. With #357 and #358 at their heads, merged together onto main 41b3d8d, the full suite (12641 passed), mypy and ruff pass.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at f6a9cb9. The history read is right: NUL framing, the pinned flags, merges, link chains and the shallow refusal hold up in the tests and in my runs. public_at can come out later than the real time, and the history counts eval_dir fixtures. Both are in the line comments.

Agreed with @napetrov on the \r in a guidance path: the history keeps it now, and fixing _guidance_at's listing in a separate change is fine.

What I ran, at f6a9cb9:

  • make ci-local --offline: 0 failed, 19 passed, 13 skipped (network and Docker jobs).
  • The CI external-catalog-smoke job (docker-oracle and docker-probe on the pinned catalog task): both pass, every isolation probe closed.
  • _read_guidance_commits on a local numpy clone (41,870 commits): 4.9 s and 19 MB of git log output for the first cell, 1.2 s of it in git.
  • A history with empty commits around a guidance commit: parsed right, and only the guidance commit is listed.

Checked, no issues

  • _link_chain split: same refusals as before (out of the tree, into eval_dir, a cycle, a dangling link), and a plain file still gives no chain.
  • guidance_latest: it is the newest commit by committer date that touched the path anywhere in the reachable history, as contract §3 says. An edit on a branch later dropped by -s ours can therefore be named, but that errs toward reporting a leak, which is the safe side.
  • Concurrency and cache: one read per key; the lock map only grows by one small entry per key.
  • Docs and changelog: contract §3 and §4.6 and the fragment describe the new fields and the shallow refusal. No scoring or repo compare change, so published numbers stay comparable.

Comment thread dokimi/dokimi/repos/replay/build.py Resolved
Comment thread dokimi/dokimi/repos/guidance_history.py Resolved
Comment thread dokimi/dokimi/repos/guidance_history.py Resolved
Comment thread dokimi/dokimi/repos/guidance_history.py Resolved
Settles the review point on intel#359 that public_at was the
minimum over only the gold threads with a usable time: a skipped thread
posted first moved it later, so a guidance commit between the two times
read as made before the answer was public.

_public_at now returns None when any gold thread's first-comment
createdAt is missing, not ISO 8601 or has no offset, and the build notes
name those threads ("no public_at: ..."). The no-offset test now expects
no public_at; a test with a missing createdAt is added. Contract 4.6 and
the changelog fragment say when the key is absent.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Settles the review point on intel#359 that the history counted
paths under eval_dir, which the arm never installs: with **/AGENTS.md, a
commit that only edited a task fixture's AGENTS.md was listed in
guidance_commits and took a place under the cap.

arm_rev_history takes eval_dir (manifest.run_paths()[0], as
_guidance_at uses), it is part of the cache and lock key, and
_counts_for skips a path within it before the glob match. A link chain
needs no filter: _link_chain refuses one that reaches into eval_dir.
The contract (section 3) and the changelog fragment say so.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Settles the review point on intel#359 that the first
guidance:rev: cell reads and parses the whole history of the
repository. The read is not narrowed with pathspecs here: git log --
<pathspec> turns on history simplification, and git's :(glob) matching
is not match_glob's, so a pre-filter needs a proof that it keeps every
commit the matcher would.

The Python side was mostly match_glob, which builds its pattern on every
call; it now runs once per distinct path of a read. On a generated
history of 42,000 commits (5-6 paths each), git log took about 1.0 s
(28.8 MB) and the parse and match 0.92 s, now 0.12 s, with the same
commits. The module docstring states the cost.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Settles the review nit on intel#359 that guidance_history built
again the isolated git environment workspace._git_env defines, and
repeated base_opener's shallow check.

source.isolated_git_env() is the one policy (PATH, LANG, TMPDIR and
SYSTEMROOT kept; no system or global config, HOME at devnull, no
prompt): workspace._git_env adds the fixed author and date to it, and
the history's reads use it. source.is_shallow(git_dir, env) is the one
check, called by base_opener and by the history's refusal. The history
passes its own environment: source's default inherits the caller's,
and a GIT_DIR there wins over -C, so the check would answer for another
repository than the log after it reads. No behaviour change: the
existing tests are unchanged, and a new one pins that a git dir git
cannot read is still a PrepError, not source's SourceError.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

Round for @vlad-perevezentsev's review, at 3f2895a (CI green):

  • 3c1f93d: public_at is left out, with a build note naming the threads, when any gold thread's time is missing, not ISO 8601, or has no offset.
  • 06f64a6: the guidance history skips paths under eval_dir.
  • bcb96c1: match_glob runs once per distinct path, and the module docstring states the read's cost. The log is not narrowed with pathspecs; the reason is in the thread.
  • 3f2895a: one isolated git env (source.isolated_git_env) and one shallow check (source.is_shallow), no behaviour change.

The \r thread on _guidance_at's listing stays open, for the separate change agreed there.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I checked 3f2895a: each point from my last review is fixed, and the new commits change nothing else. The \r in _guidance_at's listing stays for the separate change agreed in its thread.

Merge: safe now: it merges cleanly into current main, 2 commits behind, and its tests pass on the merge (main changed growth_baseline.json). Once it is in, #338 and #333 conflict with it in workspace.py and need a rebase.

What I ran, at 3f2895a:

  • make ci-local --offline: 0 failed, 19 passed, 13 skipped (network and Docker jobs).
  • The PR's tests (test_repo_guidance_history.py, test_repo_replay.py, test_repo_arms.py, test_growth_ratchet.py) on a merge with current main: 169 passed.
  • _read_guidance_commits on a local numpy clone (41,870 commits): 1.2 s of git log, then 1.5 s to parse and match, down from 3.7 s.
  • Mutations: skipping a gold thread with an unusable time again fails three public_at tests; dropping the eval_dir skip fails the new fixture test.

Checked, no issues

  • public_at (3c1f93d): absent, with a build note naming the threads, when any gold time is missing, not ISO 8601 or has no offset. Contract §4.6 and the fragment say so.
  • eval_dir (06f64a6): left out of the history as _guidance_at leaves it out of the files. It is part of the cache and lock key.
  • Shared git env (3f2895a): workspace._git_env and the history both build on source.isolated_git_env. The shallow check runs in that env, so a caller's GIT_DIR cannot make it answer for another repository. base_opener keeps its old env.
  • Pathspecs: your reason not to narrow the log yet holds. It would need --full-history and a proof that :(glob) keeps a superset of match_glob.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants