feat(graph): surface unindexed source files instead of leaving them silent

[abhinav-phi]

Resolves #163.

Problem

EXTENSION_MAP covers ten extensions, and every discovery path is built from it, so a file with any other extension is invisible everywhere. The failure mode is silence:

• a repo with no supported files builds a graph of 0 nodes with exit code 0, and mex doctor shows four green ticks;
• a mixed repo (any Svelte, Vue or Astro frontend) indexes one of three source files, reports indexedFiles: 1 with no warning, and an agent asking about a symbol that exists gets the same TARGET_NOT_FOUND a typo gets.

Approach

Count the complement at the one seam every consumer already shares. corpus-policy.ts gains unindexedExtensionHistogram: a walk with the same glob options and the same ignore list as source discovery (so a repository's own graph.ignore shapes the answer), keeping only files whose extension is recognized source but no extractor claims it.

The recognized set is checked through isSupportedSourceFile against the live extension map, so an extension stops being reported the moment an extractor lands for it — the list cannot go stale.

Reporting is bounded and best-effort by contract: a file-scan cap with an explicit truncated flag, a capped histogram, and a walk that can never fail the command it decorates.

What changes

• mex graph: prints Not indexed: N source file(s) have extensions no extractor handles: .go (2), ... after the build summary; --json gains unindexedSources (total, byExtension, truncated), absent when the histogram finds nothing, so existing consumers are unchanged.
• mex graph query / mex impact: TARGET_NOT_FOUND records gain coverage context (filesIndexed, unindexedSources) — but only when it changes the record's meaning (nothing indexed, or unindexed source files exist). Ordinary misses in a healthy repository keep their exact prior shape.
• mex doctor: a Coverage line plus a next-steps hint when recognized source files are unindexable.

Verification

• New focused tests: coverage-histogram.test.ts (recognition, noise exclusion, graph.ignore handling, truncation, entry cap).
• cli-agent.test.ts: two new tests — TARGET_NOT_FOUND stays bare on a fully covered repository, and names unindexed sources on a mixed TS/Svelte/Go repo.
• Local smoke test reproducing the issue's Repro B: mixed TS/Svelte/Go repo now reports filesIndexed: 1 + unindexedSources: {total: 2, byExtension: {".go": 1, ".svelte": 1}} on both the build output and the TARGET_NOT_FOUND record; a clean repo's output is byte-identical to before.
• npm run typecheck, npm run build pass locally; targeted suites green. (The pre-existing Windows-local failures in symlink/SHM tests reproduce identically on clean main and are unrelated.)

No identity, reconciliation, schema, or drift-semantics changes are included.

[abhinav-phi]

Rebase note: this branch now sits on current main. The only code conflict was in cli-agent.ts where #193 added the configDriftRecords(session) loop just before the TARGET_NOT_FOUND emit — both kept, in that order: the config-drift records write first, then the error record with the coverage context. The CHANGELOG.md entry moved under ## [Unreleased] → ### Added (the old section this landed in was released as 0.8.1).

CI: the first run on the rebased head failed only in test/telemetry-delivery.test.ts (expected 357.77 to be less than 150) — a flush-grace timing assertion in the 0.8.1 telemetry tests, unrelated to this diff (it touches corpus-policy, cli-graph, cli-agent's error records, doctor and test/cli-smoke only). Re-triggered clean: all checks now pass (check 22.22.0/24, hub-browser, release-performance, storage-portability macos+windows), and the abstention golden from #189/#193's timeline of changes coexists with the coverage-context tests here.

All green — ready for review.

[theyashasvipandey]

Thanks @abhinav-phi, right shape for #163: reporting rather than new extractors, bounded, and wrapped so it can never fail the command carrying it. The rebase note is accurate.

I merged current main into 8b30c9a locally (only CHANGELOG.md conflicts now) and tested that: typecheck clean, your 53 tests pass, and I ran the built CLI over five fixtures plus a 30k-file tree.

Both of #163's repros are fixed for mex graph, --json and TARGET_NOT_FOUND from where-defined / who-calls / impact. A fully supported repo stays byte-identical, node_modules / dist / graph.ignore are excluded, freshness and doctor's exit code are untouched, and candidates are checked against the live extension map so an extension drops off the moment an extractor lands.

Must fix

1. The histogram re-walks the whole repository on every call

It globs **/* synchronously in runGraph, in runDoctor, and inside targetNotFoundCoverage — so on every TARGET_NOT_FOUND from graph query and impact, the path an agent hits most. On a 30k-file tree with no unindexed sources at all:

miss:  5.3 s   ->  {"type":"error","code":"TARGET_NOT_FOUND","target":"nothingHere"}
hit:   3.2 s

It walks everything, finds nothing, discards the result, and emits exactly what it emitted before. The repos paying most are the ones the feature does nothing for.

A second problem rides along: maxUnindexedFiles: 20_000 counts every scanned file, not candidates (the build's own maxSourceFiles counts source files only), so any repo over 20k non-ignored files always truncates. My 30k tree holds 600 .go files and reports:

Not indexed: 399 source file(s) have extensions no extractor handles: .go (399)
  (count may be higher: the repository walk stopped early)

399 of 600, and which 399 depends on glob order.

The fix I'd like is to compute it once during the build and have the read paths reuse it — and I want to be honest that it isn't a small change. project_metadata is already in the shipped schema (schema.sql:288) and the build already writes through it (engine-impl.ts:535), so no schema change is needed, and storedManifestIdentity (cli-agent.ts:2229) is already the read pattern. But doctor never opens the graph store, and a stored count goes stale as soon as files change without a rebuild, so it has to hang off the freshness machinery. If you'd rather not take that on in this PR, say so and I'll split it into its own issue — but then the walk needs a much tighter bound before this merges. Either way, please cap on candidate files rather than on everything scanned.

2. graph scope still reports everything is fine

#163's Repro B, and scope is where the shipped agent guidance sends agents first. Mixed repo, 1 .ts + 1 .svelte + 1 .vue:

{"type":"health","indexedFiles":1,"okFiles":1,"partialFiles":0,"failedFiles":0,"staleFiles":[]}
{"type":"summary","matchedNodes":1,…,"status":"ok","evidenceStrength":"moderate","warnings":[],…}

Two thirds of the code is invisible and the answer is ok / moderate / no warnings. The health record already carries indexedFiles (cli-agent.ts:291), so it's the natural home. Heads-up before you write it: HealthRecord is part of AGENT_PROTOCOL_VERSION = 3 and fixtures/protocol-v3/graph-scope.jsonl freezes that output byte for byte — I'll confirm whether an additive field is fine inside v3 before you update the golden.

3. CHANGELOG

Only CHANGELOG.md conflicts now. main gained the Flask (#112) and NestJS (#98) entries plus the Hub tour under ## [0.8.2] - Unreleased → ### Added; keep all of them and add yours to the same list.

Should fix

• graph rebuild and graph refresh stay silent. printUnindexedSources is only called from runGraph, so rebuild prints no coverage line and rebuild --json has no unindexedSources key — and those are the commands doctor tells people to run after a graph problem.
• Extension noise. OTHER_KNOWN_SOURCE_EXTENSIONS includes .sql, .proto, .gradle, .ps1, .bash, .zsh, .vim. This repo trips its own check: mex doctor prints ! Coverage 1 source file(s) not indexable by any extractor, and that file is src/graph/schema.sql, our own DB schema. I'd drop the extensions no extractor will plausibly claim and keep the actual languages.
• doctor's next step points at nothing. It says "Source files above exist in the repository but are absent from the graph", but doctor prints only a count — no extensions, nothing above it. Either put the top extensions on the Coverage line (same entries array mex graph formats) or point the hint at mex graph --json.

Smaller

• TARGET_NOT_FOUND's unindexedSources omits truncated, unlike --json, so a truncated count reads as exact.
• mex impact src/Checkout.vue could say this file is unindexable rather than leaving the agent to infer it from a repo-wide count.
• vendor/ isn't ignored, so vendored Go/PHP/Ruby deps get counted — consistent with the build's ignore list, just worth knowing.

[abhinav-phi]

Implemented the build-time cache in a6ae73d/a4c2fff using existing project_metadata. Reads reuse it with bounded directory-stamp validation, without a coverage re-walk. The 30k-file regression counts all 600 Go candidates; stale, malformed, or over-budget observations do not supply exact totals. Build/refresh/rebuild output, doctor guidance, extension filtering, and the miss truncated flag are updated. The changelog conflict is resolved with the upstream entries retained.

Scope now warns and degrades for observed unsupported coverage. An invalidated cache produces an explicit unknown-coverage warning while preserving the status semantics of returned evidence. HealthRecord, protocol v3, and goldens are unchanged pending your approval of any additive health field. The optional target-specific unsupported-file message is not included.

Final-head CI is green: https://github.com/mex-memory/mex/actions/runs/35164795895 . @theyashasvipandey could you review the cache/freshness approach and confirm whether you still want the additive HealthRecord field?

[abhinav-phi]

hi the coverage cache is implemented in a4c2fff

coverage is computed once at graph build time and stored in the existing metadata

reads reuse the stored count with bounded directory stamps and never rewalk the repository

a 30k file tree now counts all 600 go candidates exactly

stale malformed or over budget observations never report an exact total

graph scope warns and degrades on observed unsupported coverage and shows an unknown coverage warning when the cache is stale

the changelog conflict is resolved with all upstream entries kept

the health record and protocol goldens stay untouched pending your call on additive fields

all ci checks are green

could you review the cache approach and let me know if you still want the additive health field