fix: stop vuln-scan tests polluting shared clean_app fixture (#83)

[Wolfvin]

Root cause

tests/test_vuln_staleness.py::TestScanVulnerabilitiesCacheInfo ran scan_vulnerabilities(fixture, offline=True) directly on the shared benchmarks/fixtures/clean_app (and vulnerable_app) directory — not on a temp copy. scan_vulnerabilities writes its OSV cache to <workspace>/.codelens/osv_cache.db, so this permanently polluted the shared fixture with benchmarks/fixtures/clean_app/.codelens/osv_cache.db.

Later, tests/test_architecture.py::TestArchitectureBasic::test_auto_scans_on_fresh_workspace copies the (now-polluted) clean_app fixture into a fresh temp dir and asserts:

assert not os.path.isdir(os.path.join(fresh_clean_app, ".codelens"))

That assertion failed intermittently — specifically when test_vuln_staleness.py had run before test_architecture.py in the same session, leaving .codelens/ behind in the fixture.

Reproduction (before fix):

PYTHONPATH=scripts python3 -m pytest tests/test_vuln_staleness.py -v   # pollutes fixture
ls benchmarks/fixtures/clean_app/.codelens/                              # osv_cache.db present
PYTHONPATH=scripts python3 -m pytest tests/test_architecture.py::TestArchitectureBasic::test_auto_scans_on_fresh_workspace -v
# FAILED: AssertionError: assert not True  (fresh_clean_app/.codelens exists)

Fix chosen and why

The root cause is a side-effect (.codelens/) being written into a fixture directory that is shared across many tests. The fix removes the side-effect at the source: the shared fixture is never written to.

Concretely, a new app_fixture_copy pytest fixture (factory pattern) is added to tests/test_vuln_staleness.py. It copies a named app fixture (clean_app / vulnerable_app) into a fresh temp dir, tracks it, and tears it down on test completion. The four affected tests in TestScanVulnerabilitiesCacheInfo are routed through this fixture, so scan_vulnerabilities now writes its .codelens/osv_cache.db into the temp copy — never into the shared fixture.

This was chosen over the alternatives explicitly rejected in the issue:

• Gitignoring .codelens/ under benchmarks/fixtures/ — rejected. It only hides the symptom; the file is still on disk and the test still fails depending on run order.
• Weakening the test_auto_scans_on_fresh_workspace assertion to tolerate a pre-existing .codelens/ — rejected. That would defeat the test's whole purpose (verifying auto-scan fires on a truly fresh workspace).

The assertion in test_auto_scans_on_fresh_workspace is intentionally left unchanged — its contract (no .codelens/ in a fresh workspace) is correct; the bug was that the fixture was pre-polluted.

Files changed

• tests/test_vuln_staleness.py
  • Added app_fixture_copy fixture (factory pattern) that copies a named app fixture into a fresh temp dir and removes it on teardown.
  • Updated the four tests in TestScanVulnerabilitiesCacheInfo (test_clean_app_no_deps_has_cache_info, test_vulnerable_app_has_stale_cache_info, test_cache_info_shape, test_cache_info_is_additive) to take app_fixture_copy and run against a temp copy instead of the shared fixture path.

No production code under scripts/ is touched. No assertions are weakened. No .gitignore changes.

Definition of Done — verification

1. Full test suite 3× consecutively (no manual cleanup between runs)

Command run 3× back-to-back (no rm in between):

PYTHONPATH=scripts python3 -m pytest tests/ -v --ignore=tests/test_integration.py -p no:cacheprovider

Note on --ignore=tests/test_integration.py: test_integration.py is a smoke suite that spawns 30+ codelens CLI subprocesses, each loading tree-sitter grammars. In this 8GB sandbox the cumulative memory growth triggers the kernel OOM-killer mid-test_integration.py (Out of memory: Killed process N (python3) total-vm:~8GB). This is a pre-existing environmental issue, not caused by this PR (which only touches tests/test_vuln_staleness.py). test_architecture.py — the file containing test_auto_scans_on_fresh_workspace — runs first alphabetically and completes cleanly well before any OOM could occur. The exact-DoD command (pytest tests/ -v, no --ignore) was also run; its log shows tests/test_architecture.py ........................ [ 2%] (24/24 pass) before pytest is later killed by OOM in test_integration.py.

Results (verbose, per-test output captured via --override-ini="addopts="):

Run	Exit	test_auto_scans_on_fresh_workspace	Summary
1	0	PASSED	840 passed, 12 skipped in 15.36s
2	0	PASSED	840 passed, 12 skipped in 15.32s
3	0	PASSED	840 passed, 12 skipped in 15.25s

Explicit per-test line from each run:

# Run 1
tests/test_architecture.py::TestArchitectureBasic::test_auto_scans_on_fresh_workspace PASSED [  0%]
# Run 2
tests/test_architecture.py::TestArchitectureBasic::test_auto_scans_on_fresh_workspace PASSED [  0%]
# Run 3
tests/test_architecture.py::TestArchitectureBasic::test_auto_scans_on_fresh_workspace PASSED [  0%]

Tail of each run:

# Run 1
======================= 840 passed, 12 skipped in 15.36s =======================
# Run 2
======================= 840 passed, 12 skipped in 15.32s =======================
# Run 3
======================= 840 passed, 12 skipped in 15.25s =======================

2. Ordered run that originally triggered the bug — test_vuln_staleness.py then test_architecture.py

PYTHONPATH=scripts python3 -m pytest tests/test_vuln_staleness.py -v -p no:cacheprovider
PYTHONPATH=scripts python3 -m pytest tests/test_architecture.py -v -p no:cacheprovider

Output:

--- test_vuln_staleness.py ---
collected 39 items
tests/test_vuln_staleness.py .......................................     [100%]
============================== 39 passed in 0.90s ==============================

--- test_architecture.py ---
collected 24 items
tests/test_architecture.py ........................                      [100%]
============================== 24 passed in 2.29s ==============================

Both files pass — including test_auto_scans_on_fresh_workspace. Before the fix, the second invocation failed on that test.

3. benchmarks/fixtures/clean_app/ stays clean

After the full 3× run + the ordered run, the fixture directory has no .codelens/:

$ git status --short benchmarks/fixtures/clean_app/ benchmarks/fixtures/vulnerable_app/
# (no output — clean)

$ ls benchmarks/fixtures/clean_app/.codelens 2>&1
ls: cannot access 'benchmarks/fixtures/clean_app/.codelens': No such file or directory

$ ls benchmarks/fixtures/vulnerable_app/.codelens 2>&1
ls: cannot access 'benchmarks/fixtures/vulnerable_app/.codelens': No such file or directory

The only working-tree change is the test file edit:

$ git status --short
 M tests/test_vuln_staleness.py

Closes

Closes #83.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud