Skip to content

fix(visualizer): bounded incremental file discovery + fix test gRPC hang - #19

Merged
SushantGautam merged 1 commit into
mainfrom
fix/visualizer-bounded-scan
Oct 8, 2026
Merged

SushantGautam merged 1 commit into
mainfrom
fix/visualizer-bounded-scan

Conversation

@SushantGautam

Copy link
Copy Markdown
Collaborator

Problem

The visualizer file-tree scan (GET /api/visualizer/api/files/) performed an unbounded os.walk over the results directory, parsing every JSON file with no limits. When --results_dir resolved to a large tree (e.g. $HOME via uvx simpleaudit-studio@latest --visualize-only --results_dir ./), the scan took minutes and the browser fetch stayed pending forever ("Loading files...").

Additionally, the full infra/tests suite consistently hung at test_api_e2e.py::APIE2ETest::test_cancel_queued_run because the test triggered a real gRPC call to the unresolvable hatchet-server:7077 (compose default). After ~50 prior tests the gRPC C-core thread pool was exhausted, so the call blocked indefinitely.

Fix

Bounded scan (infra/visualizer.py)

  • Limits: 5,000 inspected JSON files, 5s budget, depth 8, 100 MB/file — enforced during traversal, not afterward.
  • Partial results with truncated, reason, inspected, elapsed_seconds metadata; never reports a truncated tree as complete.
  • Persistent metadata index (~/.simpleaudit-studio/visualizer-index.json) keyed by (path, mtime, size) — unchanged files are not re-parsed.
  • watchfiles for incremental updates via a new /api/visualizer/api/files/changed/ endpoint.
  • Frontend (static/visualizer.html) shows a truncation warning banner and polls the changed endpoint.
  • CLI (simpleaudit_studio/cli.py) warns when results dir is home/cwd and displays scan limits.

Test fix (infra/tests/test_api_e2e.py)

  • Mock audits.views.submit_audit_run in setUp to prevent a real gRPC call. Matches the pattern used by test_monitors and test_experiments.

Tests

  • 14 new visualizer tests (file/time/depth limits, symlinks, oversized files, valid results beyond cap, metadata index, watchfiles)
  • Full suite: 995 passed, 2 skipped, 4 deselected
  • Ruff clean

Benchmark

  • Real audit-results dir: 0.188s
  • Home dir: 5.0s truncated (was: minutes)

Related: #18

The visualizer file-tree scan (GET /api/visualizer/api/files/) performed an
unbounded os.walk over the results directory, parsing every JSON file with no
limits. When --results_dir resolved to a large tree (e.g. $HOME), the scan
took minutes and the browser fetch stayed pending forever.

Bounded scan (infra/visualizer.py):
- Limits: 5,000 inspected JSON files, 5s budget, depth 8, 100 MB/file —
  enforced during traversal, not afterward.
- Partial results with truncated/reason/inspected/elapsed_seconds metadata;
  never reports a truncated tree as complete.
- Persistent metadata index (~/.simpleaudit-studio/visualizer-index.json)
  keyed by (path, mtime, size) — unchanged files are not re-parsed.
- watchfiles for incremental updates via /api/visualizer/api/files/changed/.
- Frontend shows a truncation warning banner and polls the changed endpoint.
- CLI warns when results dir is home/cwd and displays scan limits.

Test fix (infra/tests/test_api_e2e.py):
- Mock audits.views.submit_audit_run in setUp to prevent a real gRPC call
  to the unresolvable hatchet-server:7077 (compose default). After ~50
  prior tests the gRPC C-core thread pool is exhausted, so the call blocks
  indefinitely. Matches the pattern used by test_monitors and test_experiments.

Tests: 14 new visualizer tests (file/time/depth limits, symlinks, oversized
files, valid results beyond cap, metadata index, watchfiles). Full suite:
995 passed, 2 skipped, 4 deselected. Ruff clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@SushantGautam
SushantGautam merged commit 77e854d into main Oct 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant