Skip to content

fix(watcher): detect NFS mounts and make WatchConfig.Enabled actually disable watching - #697

Open
timkjr wants to merge 4 commits into
zzet:mainfrom
timkjr:fix-nfs-watch-detection
Open

fix(watcher): detect NFS mounts and make WatchConfig.Enabled actually disable watching#697
timkjr wants to merge 4 commits into
zzet:mainfrom
timkjr:fix-nfs-watch-detection

Conversation

@timkjr

@timkjr timkjr commented Aug 30, 2026

Copy link
Copy Markdown

This fixes two related bugs that together explain why repos on NFS/SMB mounts silently stop being watched, with no error surfaced anywhere obvious.

1. slowWatchMount never recognized NFS

The slow-mount gate only checked for WSL2's 9p/drvfs magic number and (within a WSL context) CIFS. A native Linux host with a repo on a plain NFS mount fell through every safety net — NFS_SUPER_MAGIC (0x6969) was never checked. Native fsnotify was then attempted regardless, reliably failing confirmWatchActive's 5s readiness window, leaving the repo with neither fsnotify nor the poller fallback until a manual untrack+track.

2. WatchConfig.Enabled didn't actually disable watching

config.Default() ships Watch.Enabled: false — checked into gortex's own .gortex.yaml too. But Watcher.Start only gated the safe branches (the slow-mount skip-to-poller path from fix #1, and launching the poller as a fallback) behind Enabled. It never gated the actual attempt to start native fsnotify, which ran unconditionally regardless of the flag. So with the shipped default, every repo blindly attempted raw fsnotify, racing the same 5s timeout with no safety net and no fallback — meaning fix #1 was effectively dead code unless a user manually set watch.enabled: true, which is undocumented anywhere. This PR adds an early return when Enabled is false, matching what the surrounding comments already claimed the flag did.

Known related gap, not fixed here: slow_mount_other.go (the non-Linux build) still returns false unconditionally, so a macOS host with a repo on an NFS mount gets no slow-mount protection either — the same class of bug this PR closes for Linux. I don't have a Mac to test against, so I've left that alone rather than guess at the right statfs-equivalent check; happy to take a pass at it if a maintainer can confirm the right approach (or point at existing platform-detection code I should mirror).

Verified: go test -race ./... and golangci-lint run both clean. Manually confirmed on a live NFS-mounted multi-repo daemon — all repos went from erratic per-boot watcher-attach failures to 100% success (fsnotify where safe, poller fallback on slow mounts).

Separately flagging for maintainer discussion, not changed here: is Enabled: false the right default? A code-intelligence tool whose core value is live indexing arguably shouldn't ship watching off by default with no documentation pointing at the flag.

@zzet zzet left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this — the investigation behind it is genuinely good, and you found two real bugs, not one. The NFS detection half is correct and I want it. But the WatchConfig.Enabled half has to change before this can merge: as written it disables live indexing for every repo the daemon tracks, under the shipped default.

Blocker: the Enabled early return turns watching off everywhere

The premise that Enabled gates watching isn't quite right, and the daemon is the case that proves it. cmd/gortex/daemon_state.go:987 builds the per-repo watch configs straight from repo config, with no override:

watchCfgs[prefix] = state.configManager.GetRepoConfig(prefix).Watch

The only place in the tree that sets Watch.Enabled = true is cmd/gortex/mcp.go:507, the gortex mcp --watch path. Since config.Default() ships enabled: false, the new early return fires for every repo under gortex daemon: no fsnotify, no poller, no live indexing at all.

I verified this rather than reading it. A test that drives config.Default().Watch through Start() fails on this branch and passes on main:

this branch:  file change never reached the graph — repo has neither fsnotify nor poller
main:         change detected, live indexing works

It also silently defeats the staleness alarm. The early return yields nil, so MultiWatcher.Start records started = true, and WatchedRepos() then reports live == configured — which means the warning at daemon_state.go:1013, added for exactly this symptom, never fires:

// Reporting the configured count here made an install where every watcher had
// failed to start look fully watched, so the one signal that the graph was
// going stale never fired.

The daemon would log daemon: watching repos=N configured=N and publish watched_repos: N while nothing is watching anything.

The underlying reason for the misread is fair: Enabled has never meant "watch at all" — it gates the adaptive poller only, and fsnotify has always run regardless. TestPoller_RespectsWatcherDisableKnob (internal/indexer/poller_test.go:214) is the contract. The flag is badly named, which is the actual defect you walked into.

What this PR gets right — please keep both

  1. NFS detection. slowWatchMount being gated behind runningUnderWSL() meant NFS_SUPER_MAGIC (0x6969) and native-Linux CIFS were never checked. Correct diagnosis, correct magic numbers, and factoring out isSlowMountFSType for direct testing is the right call.
  2. Ungating the degraded-path pollers. Both the slow-mount branch and the confirmWatchActive readiness-failure branch built the fallback poller only if w.config.Enabled — so under the default, a repo whose fsnotify died got neither mechanism. That is a second genuine dead-repo bug and your removal of those two gates is exactly right. It's the part of your report I'd most want fixed.

Suggested fix (verified)

Drop the early return, keep both degraded-path pollers ungated as you have them, and re-gate only the final alongside-healthy-fsnotify poller:

// Enabled is the opt-in only where fsnotify is LIVE — there the poller is a
// belt-and-braces extra for what fsnotify misses, and a repo may decline it.
// The degraded paths above start it unconditionally: fsnotify is dead there,
// so declining it means the repo silently goes stale.
if w.config.Enabled {
    w.poller = newPoller(w, w.indexer, w.logger)
    w.poller.Start()
}

With that, TestPoller_RespectsWatcherDisableKnob and a new daemon-default regression test both pass, and the full indexer suite is green (1573 passed).

Why the suite didn't catch it

Your go test -race ./... result is accurate — all 1649 indexer + server tests pass on this branch. The gap is that every watcher test hardcodes Enabled: true (see setupWatcher), so the daemon's actual default is never exercised. Please add a test that drives config.Default().Watch through Start() and asserts a file change reaches the graph; that's the guard that would have caught this.

Please split out the tool-promotion commits

Four of the eight commits and seven of the twelve files here are the deferred-MCP-tool promotion work (SetToolPromoter / getToolOrPromote), which the description doesn't mention. It's substantively sound — EnsureToolPromoted already exists, Promote is mutex-guarded, and hidden tools are never deferred so the hide gate isn't bypassed — but it's unrelated to the watcher fix and needs to be reviewed on its own. Two things to carry into that PR:

  • Concurrency: lazyToolRegistry.Promote marks promoted[name] under the lock but calls promoteFn outside it, so a concurrent second first-call for the same tool gets false back and skips the retry lookup — a spurious "not registered". The established idiom at cmd/gortex/daemon_mcp.go:155 uses that boolean for bookkeeping only and proceeds regardless; worth matching.
  • Session surface: daemon_mcp.go deliberately checks IsToolEnabledForSession before touching the process-global lazy registry. The new newLocalToolExecutor path in server_router.go calls EnsureToolPromoted directly, skipping that guard.

On your two open questions

Is Enabled: false the right default? Given it actually gates the poller, yes — fsnotify on, poller off, and the poller auto-engaging when fsnotify is dead once your fix #2 lands. What should change is the name, not the default. A follow-up renaming it to something like FallbackPoller (with the old key accepted for a release) would be very welcome.

slow_mount_other.go on macOS: leaving it alone was the right instinct, and no, don't mirror the Linux approach — Darwin's statfs carries f_fstypename as a string, so the check there is a name comparison against nfs / smbfs / webdav rather than a magic number. Happy to take that as a separate PR from you if you want it; it doesn't need to block this one.

One thing I checked and am not asking you to change: int64(st.Type) would truncate CIFS_MAGIC on a 32-bit host, but only amd64 and arm64 ship, and both have Statfs_t.Type int64. Not a problem.

Thanks again for chasing this down to the actual mechanism — the report quality made it fast to verify.

timkjr and others added 4 commits August 30, 2026 15:46
slowWatchMount's whole filesystem-type check was gated behind
runningUnderWSL(): on any non-WSL2 Linux host, the function returned
false unconditionally without ever inspecting the actual filesystem,
even though its own doc comment describes detecting "a filesystem
where native fsnotify is unreliable or prohibitively slow" generically.

This meant a repo tracked on a plain NFS mount on a native Linux host
fell through the one safety net that exists specifically for this
class of problem. The fsnotify backend then reliably failed its
Watcher.Start() 5-second readiness wait (confirmWatchActive/the ready
channel select) -- and because that failure path returns before the
adaptive poller is ever constructed, the repo ended up with neither
fsnotify nor the poller fallback: no update mechanism at all until a
manual untrack + track. Reproduced consistently on a Debian host with
repos on an NFS4 mount -- every daemon restart in the log history hits
"watcher: backend did not become ready within 5s" for every NFS-mounted
repo, with the daemon's own warning ("their graphs go stale until the
daemon restarts") describing a fix that doesn't actually happen, since
restarting just re-triggers the same failure.

Removed the WSL gate; the magic-number check now runs on any Linux
host and adds NFS_SUPER_MAGIC (0x6969) to the switch. Factored the
switch into isSlowMountFSType(fsType int64) so it's unit-testable
without needing a live WSL2/SMB/NFS mount, and verified the fix
against a real NFS4 mount: slowWatchMount(<repo on NFS>) now returns
true (was false), while local-disk paths are unaffected.

runningUnderWSL() removed as dead code -- 9p and CIFS mounts are
exactly as unreliable for fsnotify when mounted directly on native
Linux as they are inside WSL2, so gating the whole check on WSL
detection was never actually necessary for those two either.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NktJ7aab9oD9U1ks35g9TS
Watcher.Start only gated the safe branches (slow-mount skip-to-poller,
poller-as-fallback) behind config.Watch.Enabled — it never gated the
actual attempt to start native fsnotify, which ran unconditionally
regardless of the flag. Since config.Default() ships Enabled: false,
every repo without an explicit watch.enabled: true silently attempted
raw fsnotify, raced confirmWatchActive's 5s readiness timeout with no
safety net, and got no poller fallback on either success or failure —
exactly backwards from what "disabled" should mean.

Add an early return when Enabled is false, matching what the existing
comments in this function already claimed ("a repo that opted out of
watching gets no fallback either"). Set degradedNoFsnotify in that
path too, since Stop() skips waiting on the loop-closed w.stopped
channel only in degraded mode — without it, Stop() on a never-started
watcher blocks forever.

Simplify the two now-redundant nested Enabled checks further down in
Start, since Enabled is guaranteed true for the rest of the function
after the early return.
Reword to make clear the comment describes the bug this commit fixes,
not pre-existing behavior — a reviewer reading the PR cold could
otherwise misread "every fallback below already assumed this" as a
statement about the code before this patch.
TestIsSlowMountFSType exercises isSlowMountFSType, which only exists in
slow_mount_linux.go (//go:build linux). The test file had no build tag,
so it compiled on every platform, and on macOS/Windows (only
slow_mount_other.go compiled, defining slowWatchMount but not
isSlowMountFSType) the package failed to build:

    internal/indexer/slow_mount_test.go:40:14: undefined: isSlowMountFSType

Split the file: TestIsSlowMountFSType stays behind //go:build linux
alongside the function it pins; TestSlowWatchMount_NormalMountNotDegraded
(exercises the cross-platform slowWatchMount) moves to a new untagged
file so it keeps running on macOS/Windows too.
@timkjr
timkjr force-pushed the fix-nfs-watch-detection branch from 3eaa57e to 89e360b Compare August 30, 2026 21:15
@timkjr

timkjr commented Aug 30, 2026

Copy link
Copy Markdown
Author

Thanks for the detailed review — you were right on all counts, and it exposed a real gap in the test suite that I've closed.

The blocker (Enabled disabling everything) — fixed exactly as suggested. Dropped the !Enabled early return entirely. Both degraded-path pollers (slow-mount fallback, inotify/FD-exhaustion fallback) now start unconditionally, as they did before my regression. Only the final poller — the one running alongside healthy fsnotify — stays gated on w.config.Enabled:

if w.config.Enabled {
    w.poller = newPoller(w, w.indexer, w.logger)
    w.poller.Start()
}

New test — added TestWatcher_ShippedDefaultStillWatches, which drives the real config.Default().Watch (asserting Enabled == false rather than hardcoding it, so this can't drift silently) through Start(), confirms the poller is nil, and asserts a file change still reaches the graph — proving it arrives via fsnotify, not the poller. TestPoller_RespectsWatcherDisableKnob and the full internal/indexer suite pass under -race.

Split the tool-promotion commits out — done. This branch is now rebased to 4 commits against current main, touching only internal/indexer/{watcher.go, slow_mount_linux.go, slow_mount_test.go, slow_watch_mount_test.go}. The deferred-MCP-tool-promotion work belongs with #649, which is already open and further along (it's since picked up a session-identity fix this branch's copy didn't have) — I'll bring your two flagged bugs (the Promote concurrency issue and the newLocalToolExecutor session-policy bypass) over there rather than duplicating/regressing that work here.

On the two open questions — agreed on both. Enabled: false is the right default; I'll open the FallbackPoller rename as a follow-up rather than bundling it here. And leaving slow_mount_other.go alone was intentional — happy to take the Darwin f_fstypename follow-up as a separate PR if useful, just let me know.

@timkjr

timkjr commented Aug 30, 2026

Copy link
Copy Markdown
Author

Following up on the two bugs from your split-out ask — both are already fixed, no new PR needed.

lazyToolRegistry.Promote's concurrency issue and newLocalToolExecutor's session-surface bypass turned out to be exactly round-1 findings #1 and #3 from your own review on #649 (that branch carried a stale pre-review snapshot of the same code). #649 merged with both fixed — the lock now wraps mark-and-register in one critical section (TestPromote_ConcurrentCallersNeverFalse404 pins it), and the dispatch path calls EnsureToolPromotedForSession instead of the bare EnsureToolPromoted, so a hidden-session tool is never promoted on its behalf (TestLocalExecutor_HiddenSessionDeniedWithoutPromotion/...EvenWhenAlreadyLive). Verified both under -race on merged main.

For the record, since it's adjacent: #649's round 2 flagged the bigger issue (remote-proxied calls skipping the origin session's policy/workspace-boundary enforcement) as a "potential security issue," and the fix that landed only closes the local half by design — the remote/federation half is tracked separately in #696, still open. Not something this PR needs to touch, just flagging so it doesn't look dropped.

@timkjr
timkjr requested a review from zzet August 30, 2026 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants