From 2a7e0cb709f02c7be015f27752a605b945ac8c5b Mon Sep 17 00:00:00 2001
From: Jason Pascucci <jpascucci@nvidia.com>
Date: Mon, 19 Oct 2026 08:23:33 -0700
Subject: [PATCH] Remove flaky
 test_concurrent_eviction_does_not_delete_replaced_file
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Refs #2745

The failure is a race in the test’s shutdown protocol, not clear evidence of a cache bug.
A valid failing interleaving is:
1. The rewriter writes survivor.
2. The churner’s final size-cap pass legitimately evicts survivor; the 5,000-byte cap only holds two ~2,000-byte entries.
3. The churner exits.
4. The parent sets done_event.
5. The rewriter observes the event before its next write and exits.
6. The final lookup returns None.

The stat guard only prevents deletion when the file changes between the eviction snapshot and its second stat(). It does not protect survivor from ordinary LRU eviction. There is also an explicitly documented residual race between the second stat() and unlink().

Supporting evidence:
- Both subprocesses exited successfully; only the final survival assertion failed.
- Repetitions 1–3 and 5 passed; repetition 4 failed under Python 3.14t with parallel tests and --count=5.
- The same failure was previously reported on an unrelated PR and identified as racy.
- A deterministic test already forces replacement between the scan and re-stat and verifies the intended guard at [test_program_cache.py (line 1697)](/data/src/cuda-python/cuda_core/tests/test_program_cache.py:1697).

I recommend removing the stochastic multiprocess test at [test_program_cache_multiprocess.py (line 148)](/data/src/cuda-python/cuda_core/tests/test_program_cache_multiprocess.py:148), since the deterministic test covers the property reliably. Simply serializing it or adding a final rewrite would only hide the race, not validate the stat guard.
---
 .../tests/test_program_cache_multiprocess.py  | 67 -------------------
 1 file changed, 67 deletions(-)

diff --git a/cuda_core/tests/test_program_cache_multiprocess.py b/cuda_core/tests/test_program_cache_multiprocess.py
index f73256266da..329229425ef 100644
--- a/cuda_core/tests/test_program_cache_multiprocess.py
+++ b/cuda_core/tests/test_program_cache_multiprocess.py
@@ -123,70 +123,3 @@ def test_concurrent_reader_never_sees_torn_file(tmp_path):
     # mismatches.
     assert hits == 200
     assert mismatches == 0
-
-
-def _worker_size_cap_writer(root: str, prefix: bytes, payload: bytes, count: int, max_size_bytes: int) -> None:
-    """Hammer a small-cap cache with churning writes so eviction fires often."""
-    from cuda.core.utils import FileStreamProgramCache
-
-    with FileStreamProgramCache(root, max_size_bytes=max_size_bytes) as cache:
-        for i in range(count):
-            cache[prefix + str(i).encode()] = payload
-
-
-def _worker_size_cap_rewriter(root: str, key: bytes, payload: bytes, max_size_bytes: int, done_event) -> None:
-    """Repeatedly rewrite ``key`` with a fresh value until ``done_event`` fires."""
-    from cuda.core.utils import FileStreamProgramCache
-
-    with FileStreamProgramCache(root, max_size_bytes=max_size_bytes) as cache:
-        i = 0
-        while not done_event.is_set():
-            cache[key] = payload + str(i).encode()
-            i += 1
-
-
-def test_concurrent_eviction_does_not_delete_replaced_file(tmp_path):
-    """Eviction is stat-guarded: while one process is evicting an entry to
-    bring the cache under its size cap, another process may have already
-    ``os.replace``-d a fresh value into the same path. The evictor must
-    refuse to unlink in that case, otherwise the racing rewriter's
-    just-committed entry vanishes.
-
-    The rewriter does NO post-race uncontested write -- the entry's
-    survival has to come from the in-race rewrites being preserved by
-    the stat-guarded eviction path."""
-    from cuda.core.utils import FileStreamProgramCache
-
-    root = str(tmp_path / "fc")
-    payload = b"X" * 2000
-    cap = 5000  # fits 2 raw 2000B entries; the third write triggers eviction
-
-    ctx = _mp.get_context("spawn")
-    done_event = ctx.Event()
-    rewriter = ctx.Process(
-        target=_worker_size_cap_rewriter,
-        args=(root, b"survivor", payload, cap, done_event),
-    )
-    # Churning writer creates new keys faster than eviction can drain them,
-    # forcing _enforce_size_cap to consider 'survivor' for eviction many times.
-    churner = ctx.Process(
-        target=_worker_size_cap_writer,
-        args=(root, b"churn-", payload, 80, cap),
-    )
-    rewriter.start()
-    churner.start()
-    churner.join(timeout=60)
-    done_event.set()
-    rewriter.join(timeout=60)
-    assert rewriter.exitcode == 0
-    assert churner.exitcode == 0
-
-    # If eviction blindly unlinked replaced files, the rewriter's last
-    # in-race write would be gone -- there is no post-race write to mask
-    # the failure. The value must also start with ``payload`` (not a
-    # ``churn-`` value) because each rewriter iteration writes
-    # ``payload + str(i).encode()``.
-    with FileStreamProgramCache(root, max_size_bytes=cap) as cache:
-        got = cache.get(b"survivor")
-        assert got is not None, "rewriter's entry was evicted by racing churner"
-        assert got.startswith(payload), f"survivor value not from rewriter: {got[:32]!r}..."