Repository navigation
FIX: Reread recently modified scorer metrics instead of caching them - #3697
Merged
Roman Lutz (romanlutz) merged 10 commits intoNov 1, 2026
Merged
Roman Lutz (romanlutz) merged 10 commits into
Roman Lutz (romanlutz) merged 10 commits into
Conversation
The scenario progress read model keyed its scorer-metrics cache on the file's (mtime_ns, ctime_ns, size, inode). Filesystems update timestamps at the clock tick, so a same-length rewrite within one tick kept the key unchanged and the GUI kept serving the old metrics. Treat metadata as proof the file is unchanged only once its mtime is older than a 2-second window; until then, read without caching. Backdate the metrics file in the cache-reuse test so it no longer depends on timing, and add regression tests for both sides of the window. Fixes microsoft#3696
Contributor
Author
|
@microsoft-github-policy-service agree |
Roman Lutz (romanlutz)
approved these changes
Nov 1, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Roman Lutz (romanlutz)
added a commit
to FazeelUsmani/PyRIT
that referenced
this pull request
Nov 1, 2026
Integrate microsoft#3770, microsoft#3697 and microsoft#3695 normally. As requested, retain the upstream deterministic one-node mapping-budget assertion, truncation and exact totals; do not restore the removed wall-clock benchmark. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #3696.
ScenarioProgressReadModel._load_objective_metricscaches scorer metrics under the metrics file's(st_mtime_ns, st_ctime_ns, st_size, st_ino). Filesystems update timestamps at the clock tick, so a same-length rewrite within one tick leaves the key unchanged. The GUI then keeps showing the old metrics. The same collision makestest_progress_reuses_metrics_until_file_changes_even_for_finished_runsflaky (assert 3 == 4).This applies the racy-timestamp rule Git uses for its index: matching metadata proves the file is unchanged only once its mtime is older than the timestamp resolution. While the mtime is within
_RECENT_FILE_WINDOW_NS(2 s, which covers FAT's resolution) of the current time, the metrics are read without caching. After that, caching works as before, so unchanged files are still never re-read and the I/O savings from #3348 are kept. The uncached read moved into_read_objective_metrics, which the cached loader also calls.Tests and Documentation
test_progress_reuses_metrics_until_file_changes_even_for_finished_runsnow backdates each write withos.utime, so cache hits and misses no longer depend on timing. 30 repeats pass; onmainit failed about half the time when run on its own.test_load_objective_metrics_rereads_recent_file_with_unchanged_metadatamakesPath.statreturn the same metadata across a same-length rewrite. It fails 10/10 without the fix (assert 0.75 == 0.85) and passes with it.test_load_objective_metrics_reuses_cache_once_file_is_older_than_recent_windowchecks that a file older than the window is still served from the cache.tests/unit/backend: 3,313 passed.ruff checkandruff formatare clean, andtyreports no new diagnostics on the changed files.No documentation changes; the behavior is internal to the GUI backend.