fix(watcher): detect NFS mounts and make WatchConfig.Enabled actually disable watching - #697
fix(watcher): detect NFS mounts and make WatchConfig.Enabled actually disable watching#697timkjr wants to merge 4 commits into
Conversation
zzet
left a comment
There was a problem hiding this comment.
Thanks for this — the investigation behind it is genuinely good, and you found two real bugs, not one. The NFS detection half is correct and I want it. But the WatchConfig.Enabled half has to change before this can merge: as written it disables live indexing for every repo the daemon tracks, under the shipped default.
Blocker: the Enabled early return turns watching off everywhere
The premise that Enabled gates watching isn't quite right, and the daemon is the case that proves it. cmd/gortex/daemon_state.go:987 builds the per-repo watch configs straight from repo config, with no override:
watchCfgs[prefix] = state.configManager.GetRepoConfig(prefix).WatchThe only place in the tree that sets Watch.Enabled = true is cmd/gortex/mcp.go:507, the gortex mcp --watch path. Since config.Default() ships enabled: false, the new early return fires for every repo under gortex daemon: no fsnotify, no poller, no live indexing at all.
I verified this rather than reading it. A test that drives config.Default().Watch through Start() fails on this branch and passes on main:
this branch: file change never reached the graph — repo has neither fsnotify nor poller
main: change detected, live indexing works
It also silently defeats the staleness alarm. The early return yields nil, so MultiWatcher.Start records started = true, and WatchedRepos() then reports live == configured — which means the warning at daemon_state.go:1013, added for exactly this symptom, never fires:
// Reporting the configured count here made an install where every watcher had
// failed to start look fully watched, so the one signal that the graph was
// going stale never fired.The daemon would log daemon: watching repos=N configured=N and publish watched_repos: N while nothing is watching anything.
The underlying reason for the misread is fair: Enabled has never meant "watch at all" — it gates the adaptive poller only, and fsnotify has always run regardless. TestPoller_RespectsWatcherDisableKnob (internal/indexer/poller_test.go:214) is the contract. The flag is badly named, which is the actual defect you walked into.
What this PR gets right — please keep both
- NFS detection.
slowWatchMountbeing gated behindrunningUnderWSL()meantNFS_SUPER_MAGIC(0x6969) and native-Linux CIFS were never checked. Correct diagnosis, correct magic numbers, and factoring outisSlowMountFSTypefor direct testing is the right call. - Ungating the degraded-path pollers. Both the slow-mount branch and the
confirmWatchActivereadiness-failure branch built the fallback poller onlyif w.config.Enabled— so under the default, a repo whose fsnotify died got neither mechanism. That is a second genuine dead-repo bug and your removal of those two gates is exactly right. It's the part of your report I'd most want fixed.
Suggested fix (verified)
Drop the early return, keep both degraded-path pollers ungated as you have them, and re-gate only the final alongside-healthy-fsnotify poller:
// Enabled is the opt-in only where fsnotify is LIVE — there the poller is a
// belt-and-braces extra for what fsnotify misses, and a repo may decline it.
// The degraded paths above start it unconditionally: fsnotify is dead there,
// so declining it means the repo silently goes stale.
if w.config.Enabled {
w.poller = newPoller(w, w.indexer, w.logger)
w.poller.Start()
}With that, TestPoller_RespectsWatcherDisableKnob and a new daemon-default regression test both pass, and the full indexer suite is green (1573 passed).
Why the suite didn't catch it
Your go test -race ./... result is accurate — all 1649 indexer + server tests pass on this branch. The gap is that every watcher test hardcodes Enabled: true (see setupWatcher), so the daemon's actual default is never exercised. Please add a test that drives config.Default().Watch through Start() and asserts a file change reaches the graph; that's the guard that would have caught this.
Please split out the tool-promotion commits
Four of the eight commits and seven of the twelve files here are the deferred-MCP-tool promotion work (SetToolPromoter / getToolOrPromote), which the description doesn't mention. It's substantively sound — EnsureToolPromoted already exists, Promote is mutex-guarded, and hidden tools are never deferred so the hide gate isn't bypassed — but it's unrelated to the watcher fix and needs to be reviewed on its own. Two things to carry into that PR:
- Concurrency:
lazyToolRegistry.Promotemarkspromoted[name]under the lock but callspromoteFnoutside it, so a concurrent second first-call for the same tool getsfalseback and skips the retry lookup — a spurious "not registered". The established idiom atcmd/gortex/daemon_mcp.go:155uses that boolean for bookkeeping only and proceeds regardless; worth matching. - Session surface:
daemon_mcp.godeliberately checksIsToolEnabledForSessionbefore touching the process-global lazy registry. The newnewLocalToolExecutorpath inserver_router.gocallsEnsureToolPromoteddirectly, skipping that guard.
On your two open questions
Is Enabled: false the right default? Given it actually gates the poller, yes — fsnotify on, poller off, and the poller auto-engaging when fsnotify is dead once your fix #2 lands. What should change is the name, not the default. A follow-up renaming it to something like FallbackPoller (with the old key accepted for a release) would be very welcome.
slow_mount_other.go on macOS: leaving it alone was the right instinct, and no, don't mirror the Linux approach — Darwin's statfs carries f_fstypename as a string, so the check there is a name comparison against nfs / smbfs / webdav rather than a magic number. Happy to take that as a separate PR from you if you want it; it doesn't need to block this one.
One thing I checked and am not asking you to change: int64(st.Type) would truncate CIFS_MAGIC on a 32-bit host, but only amd64 and arm64 ship, and both have Statfs_t.Type int64. Not a problem.
Thanks again for chasing this down to the actual mechanism — the report quality made it fast to verify.
slowWatchMount's whole filesystem-type check was gated behind
runningUnderWSL(): on any non-WSL2 Linux host, the function returned
false unconditionally without ever inspecting the actual filesystem,
even though its own doc comment describes detecting "a filesystem
where native fsnotify is unreliable or prohibitively slow" generically.
This meant a repo tracked on a plain NFS mount on a native Linux host
fell through the one safety net that exists specifically for this
class of problem. The fsnotify backend then reliably failed its
Watcher.Start() 5-second readiness wait (confirmWatchActive/the ready
channel select) -- and because that failure path returns before the
adaptive poller is ever constructed, the repo ended up with neither
fsnotify nor the poller fallback: no update mechanism at all until a
manual untrack + track. Reproduced consistently on a Debian host with
repos on an NFS4 mount -- every daemon restart in the log history hits
"watcher: backend did not become ready within 5s" for every NFS-mounted
repo, with the daemon's own warning ("their graphs go stale until the
daemon restarts") describing a fix that doesn't actually happen, since
restarting just re-triggers the same failure.
Removed the WSL gate; the magic-number check now runs on any Linux
host and adds NFS_SUPER_MAGIC (0x6969) to the switch. Factored the
switch into isSlowMountFSType(fsType int64) so it's unit-testable
without needing a live WSL2/SMB/NFS mount, and verified the fix
against a real NFS4 mount: slowWatchMount(<repo on NFS>) now returns
true (was false), while local-disk paths are unaffected.
runningUnderWSL() removed as dead code -- 9p and CIFS mounts are
exactly as unreliable for fsnotify when mounted directly on native
Linux as they are inside WSL2, so gating the whole check on WSL
detection was never actually necessary for those two either.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NktJ7aab9oD9U1ks35g9TS
Watcher.Start only gated the safe branches (slow-mount skip-to-poller,
poller-as-fallback) behind config.Watch.Enabled — it never gated the
actual attempt to start native fsnotify, which ran unconditionally
regardless of the flag. Since config.Default() ships Enabled: false,
every repo without an explicit watch.enabled: true silently attempted
raw fsnotify, raced confirmWatchActive's 5s readiness timeout with no
safety net, and got no poller fallback on either success or failure —
exactly backwards from what "disabled" should mean.
Add an early return when Enabled is false, matching what the existing
comments in this function already claimed ("a repo that opted out of
watching gets no fallback either"). Set degradedNoFsnotify in that
path too, since Stop() skips waiting on the loop-closed w.stopped
channel only in degraded mode — without it, Stop() on a never-started
watcher blocks forever.
Simplify the two now-redundant nested Enabled checks further down in
Start, since Enabled is guaranteed true for the rest of the function
after the early return.
Reword to make clear the comment describes the bug this commit fixes, not pre-existing behavior — a reviewer reading the PR cold could otherwise misread "every fallback below already assumed this" as a statement about the code before this patch.
TestIsSlowMountFSType exercises isSlowMountFSType, which only exists in
slow_mount_linux.go (//go:build linux). The test file had no build tag,
so it compiled on every platform, and on macOS/Windows (only
slow_mount_other.go compiled, defining slowWatchMount but not
isSlowMountFSType) the package failed to build:
internal/indexer/slow_mount_test.go:40:14: undefined: isSlowMountFSType
Split the file: TestIsSlowMountFSType stays behind //go:build linux
alongside the function it pins; TestSlowWatchMount_NormalMountNotDegraded
(exercises the cross-platform slowWatchMount) moves to a new untagged
file so it keeps running on macOS/Windows too.
3eaa57e to
89e360b
Compare
|
Thanks for the detailed review — you were right on all counts, and it exposed a real gap in the test suite that I've closed. The blocker ( if w.config.Enabled {
w.poller = newPoller(w, w.indexer, w.logger)
w.poller.Start()
}New test — added Split the tool-promotion commits out — done. This branch is now rebased to 4 commits against current On the two open questions — agreed on both. |
|
Following up on the two bugs from your split-out ask — both are already fixed, no new PR needed.
For the record, since it's adjacent: #649's round 2 flagged the bigger issue (remote-proxied calls skipping the origin session's policy/workspace-boundary enforcement) as a "potential security issue," and the fix that landed only closes the local half by design — the remote/federation half is tracked separately in #696, still open. Not something this PR needs to touch, just flagging so it doesn't look dropped. |
This fixes two related bugs that together explain why repos on NFS/SMB mounts silently stop being watched, with no error surfaced anywhere obvious.
1.
slowWatchMountnever recognized NFSThe slow-mount gate only checked for WSL2's 9p/drvfs magic number and (within a WSL context) CIFS. A native Linux host with a repo on a plain NFS mount fell through every safety net —
NFS_SUPER_MAGIC(0x6969) was never checked. Native fsnotify was then attempted regardless, reliably failingconfirmWatchActive's 5s readiness window, leaving the repo with neither fsnotify nor the poller fallback until a manual untrack+track.2.
WatchConfig.Enableddidn't actually disable watchingconfig.Default()shipsWatch.Enabled: false— checked into gortex's own.gortex.yamltoo. ButWatcher.Startonly gated the safe branches (the slow-mount skip-to-poller path from fix #1, and launching the poller as a fallback) behindEnabled. It never gated the actual attempt to start native fsnotify, which ran unconditionally regardless of the flag. So with the shipped default, every repo blindly attempted raw fsnotify, racing the same 5s timeout with no safety net and no fallback — meaning fix #1 was effectively dead code unless a user manually setwatch.enabled: true, which is undocumented anywhere. This PR adds an early return whenEnabledis false, matching what the surrounding comments already claimed the flag did.Known related gap, not fixed here:
slow_mount_other.go(the non-Linux build) still returnsfalseunconditionally, so a macOS host with a repo on an NFS mount gets no slow-mount protection either — the same class of bug this PR closes for Linux. I don't have a Mac to test against, so I've left that alone rather than guess at the rightstatfs-equivalent check; happy to take a pass at it if a maintainer can confirm the right approach (or point at existing platform-detection code I should mirror).Verified:
go test -race ./...andgolangci-lint runboth clean. Manually confirmed on a live NFS-mounted multi-repo daemon — all repos went from erratic per-boot watcher-attach failures to 100% success (fsnotify where safe, poller fallback on slow mounts).Separately flagging for maintainer discussion, not changed here: is
Enabled: falsethe right default? A code-intelligence tool whose core value is live indexing arguably shouldn't ship watching off by default with no documentation pointing at the flag.