BUG: Scenario progress serves stale scorer metrics after a same-length rewrite within one timestamp tick #3696

Describe the bug

ScenarioProgressReadModel._load_objective_metrics in pyrit/backend/services/scenario_progress_read_model.py caches the parsed scorer metrics under the file's (st_mtime_ns, st_ctime_ns, st_size, st_ino), assuming any rewrite of objective_achieved_metrics.jsonl changes at least one of them. That doesn't hold: most filesystems update timestamps at the kernel clock tick (a few milliseconds on Linux), not per nanosecond. A rewrite of the same length within one tick keeps the same inode, size, mtime and ctime. The cache then keeps returning the old metrics, and the GUI shows outdated accuracy, F1, precision and recall until the file changes again. There's no error or log.

The same collision makes test_progress_reuses_metrics_until_file_changes_even_for_finished_runs in tests/unit/backend/test_scenario_progress_read_model.py flaky. The test deletes the file and recreates it with identical content. The new file reuses the inode within the same tick, so the lookup hits the cache and assert load_metrics.call_count == 4 fails with assert 3 == 4. For me it fails about half the time when run on its own.

Steps/Code to Reproduce

import json
import tempfile
from pathlib import Path
from unittest.mock import patch

from pyrit.backend.services import scenario_progress_read_model as progress_module
from pyrit.backend.services.scenario_progress_read_model import ScenarioProgressReadModel
from pyrit.models import ComponentIdentifier
from pyrit.models.identifiers.evaluation_identifier import ScorerEvaluationIdentifier

eval_hash = ScorerEvaluationIdentifier(ComponentIdentifier(class_name="S", class_module="m")).eval_hash
evals_dir = Path(tempfile.mkdtemp())
metrics_file = evals_dir / "objective" / "objective_achieved_metrics.jsonl"
metrics_file.parent.mkdir()


def write(accuracy: float) -> None:
    metrics = {"num_responses": 100, "num_human_raters": 3, "accuracy": accuracy,
               "accuracy_standard_error": 0.01, "f1_score": accuracy, "precision": accuracy, "recall": accuracy}
    metrics_file.write_text(json.dumps({"eval_hash": eval_hash, "metrics": metrics}) +"\n", encoding="utf-8")


stale = 0
trials = 200
with patch.object(progress_module, "SCORER_EVALS_PATH", evals_dir):
    for _ in range(trials):
        ScenarioProgressReadModel._load_unchanged_objective_metrics.cache_clear()
        write(0.75)
        ScenarioProgressReadModel._load_objective_metrics(eval_hash=eval_hash)  # a poll caches 0.75
        write(0.85)  # same length, so st_size doesn't change
        if ScenarioProgressReadModel._load_objective_metrics(eval_hash=eval_hash).accuracy != 0.85:
            stale += 1

print(f"stale metrics returned in {stale}/{trials} polls after the file was rewritten")

Expected Results

stale metrics returned in 0/200 polls after the file was rewritten

Actual Results

stale metrics returned in 197/200 polls after the file was rewritten (196 to 197 across runs)

The flaky test, run on its own:

E           AssertionError: assert 3 == 4
E            +  where 3 = <MagicMock name='find_objective_metrics_by_eval_hash' ...>.call_count
tests/unit/backend/test_scenario_progress_read_model.py:189: AssertionError

Values of different lengths (for example 0.75 and 0.9) don't reproduce it, becausest_size changes.

Proposed fix

Use the "racy timestamp" rule Git applies to its index. A metadata match only proves the file is unchanged if the file was last modified more than one timestamp tick before the stat. While the mtime is still within a short window of the current time, skip the cache and read the file. After that, cache as now.

_RACY_MTIME_WINDOW_NS = 2_000_000_000  # covers coarse (e.g. 2 s FAT) timestamp granularity

try:
    file_stat = file_path.stat()
except FileNotFoundError:
    file_version = None
else:
    if time.time_ns() - file_stat.st_mtime_ns < _RACY_MTIME_WINDOW_NS:
        # A rewrite later in this same tick would keep this metadata, so don't cache it yet.
        return ScenarioProgressReadModel._load_unchanged_objective_metrics.__wrapped__(
            eval_hash=eval_hash, file_path=file_path, _file_version=None
        )
    file_version = (file_stat.st_mtime_ns, file_stat.st_ctime_ns, file_stat.st_size, file_stat.st_ino)

The only extra cost is reading the file on polls in the first couple of seconds after it changes. Unchanged files are still never re-read, so the I/O savings from #3348 are kept. With this change, the script above reports 0/200.

The flaky test should keep asserting that a delete-and-recreate triggers a reload. Fora deterministic regression test, patch time.time_ns (or the file's mtime via os.utime) instead of depending on wall-clock timing:

• A same-length rewrite while the mtime is recent returns the new metrics.
• Once the mtime is older than the window, an unchanged file hits the cache.

I'm happy to open a PR with the fix and the regression tests.

Versions

• OS: Linux 5.15.167.4-microsoft-standard-WSL2 (Ubuntu, ext4)
• Python version: 3.12.3
• PyRIT version: 1.2.0.dev0, installed from main (d3446e0) in editable mode
• Dependencies: synced to uv.lock with uv sync --frozen --group dev

Current coverage, October 31, 2026

Assigned to WaqarAnwar and covered by open #3697, with a proposed recent-file uncached read and regression tests. Continue/review that existing work rather than open another metrics-cache fix. This is freshness maintenance for delivered progress/scorer metrics, not a new Scenario GUI milestone, accounting policy, or result-table feature.

Preserve both correctness on same-length/metadata-colliding rewrites and caching for unchanged settled files. The two-second window and reproduction/test results above and in #3697 are the reporter/author's proposal and evidence, not independently executed or adopted by this roadmap audit.