Fix git sync for non-ASCII filenames: deletions never committed, conflicts misrouted#458
Open
hannut wants to merge 1 commit into
Open
Fix git sync for non-ASCII filenames: deletions never committed, conflicts misrouted#458hannut wants to merge 1 commit into
hannut wants to merge 1 commit into
Conversation
…licts resolve, and commit messages show the real filename.
Git C-quotes non-ASCII filenames in its default output
("hyv\303\244\303\244 y\303\266t\303\244.md"), and the server parsed
that escaped string as a literal path. Reused as a pathspec it matches
nothing, so deleting a file with e.g. Nordic umlauts in its name was
never committed — every sync cycle concluded there was nothing to do.
The same parsing fed conflict detection (misrouting content conflicts
into the non-content auto-resolve path, which then aborted the merge
and paused sync), conflict-store reconciliation, dirty-overlap checks
before branch switches, skill version restore, and auto-save commit
messages.
All git path-listing calls now pass -z and split on NUL via a shared
splitNulSeparatedPaths helper, so paths round-trip as raw UTF-8.
The git-dirty porcelain parser reads the -z record format, where a
rename's original path arrives as the following NUL record.
|
Thanks for the contribution! Your PR could not be merged automatically: it overlaps other changes that aren't visible here, so a maintainer needs to reconcile it by hand. No action is needed from you. Your PR is already based on the latest A maintainer will resolve it and land your change; this PR will close automatically once it merges. This comment will be updated as the status changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Deleting a file whose name contains non-ASCII characters (umlauts, accents, CJK — e.g.
hyvää yötä.md) is never committed by the sync engine. The deletion silently fails to sync on every push cycle, forever; a restart or manual re-sync doesn't help. The only workaround is a manualgit rm+ commit.Root cause
Git C-quotes non-ASCII filenames in its default output (
core.quotepath=true):SyncEngine.listHeadContentPaths()parses that escaped string as a literal path and later feeds it togit rm --cached --as a pathspec, which matches nothing — sowrite-treeproduces a tree identical to HEAD and the engine concludes there's nothing to commit.The same quoted-output parsing exists in several other places with the same failure mode:
handleMergeConflict()— a conflict on a non-ASCII-named file gets the escaped name, which fails the.mddoc-file check (the escaped form has a trailing"), misroutes the file into the non-content auto-resolve path, and thegit checkout --theirs -- <escaped>there fails → merge aborted, sync paused, instead of surfacing the conflict.sync-engine.ts,boot.ts,conflict-storage.ts) — escaped names stored/compared, and resolve strategies reuse them asgit rm/git addpathspecs.dirtyFilesOverlapWith()(git-dirty.ts) — escaped names in the user-facing overlap list.restoreSkillVersion()(skill-restore.ts) — escapedls-treeoutput fed togit show, so skills containing non-ASCII-named files can't be restored.diff-treeoutput used for auto-save commit messages — garbled names in the message.Fix
All git path-listing invocations (
ls-tree,diff-tree,diff --name-only,diff-index,status --porcelain) now pass-zand split on NUL via a sharedsplitNulSeparatedPathshelper ingit-handle.ts, so paths round-trip as raw UTF-8 with no quoting to undo. Thegit-dirty.tsporcelain parser was updated for the-zrecord format (a rename's original path arrives as the following NUL record).These sites change together deliberately: conflict paths written by the sync engine are consumed by
conflict-storage.tsandboot.ts, so fixing only the writer would make real-vs-escaped comparisons wrongly clear live conflict entries.Note for existing installs:
conflicts.jsonentries persisted by older versions in escaped form are pruned at the next reconcile, and the existing stale-MERGE_HEAD handling recovers the merge state from git itself.Tests
Written first and confirmed failing before the fix, passing after (their ASCII twins pass on main):
sync-engine.test.ts: deleting a committed non-ASCII-named file commits and pushes, with the real filename in the commit message; a remote-modify/local-delete conflict on a non-ASCII-named file surfaces as a content conflict with the real UTF-8 path.git-dirty.test.ts: non-ASCII overlap reported with its real name; staged rename reports the new path (pins the-zporcelain record format).bun run lintandbun run typecheckpass. The server suite passes apart from three failures that are unrelated and reproduce identically on a pristinemaincheckout in my environment (build-skill-bundles_sharedasset guard, desktop IPC ratchet marker, CLI bundled jsonc write).A changeset is included. Also manually verified on a real knowledge base running this build: deleting a file with umlauts in its name now commits and syncs.