Skip to content

Fix git sync for non-ASCII filenames: deletions never committed, conflicts misrouted#458

Open
hannut wants to merge 1 commit into
inkeep:mainfrom
hannut:fix/non-ascii-filename-sync
Open

Fix git sync for non-ASCII filenames: deletions never committed, conflicts misrouted#458
hannut wants to merge 1 commit into
inkeep:mainfrom
hannut:fix/non-ascii-filename-sync

Conversation

@hannut

@hannut hannut commented Jul 4, 2026

Copy link
Copy Markdown

Problem

Deleting a file whose name contains non-ASCII characters (umlauts, accents, CJK — e.g. hyvää yötä.md) is never committed by the sync engine. The deletion silently fails to sync on every push cycle, forever; a restart or manual re-sync doesn't help. The only workaround is a manual git rm + commit.

Root cause

Git C-quotes non-ASCII filenames in its default output (core.quotepath=true):

$ git ls-tree -r --name-only HEAD | grep yötä
"hyv\303\244\303\244 y\303\266t\303\244.md"

SyncEngine.listHeadContentPaths() parses that escaped string as a literal path and later feeds it to git rm --cached -- as a pathspec, which matches nothing — so write-tree produces a tree identical to HEAD and the engine concludes there's nothing to commit.

The same quoted-output parsing exists in several other places with the same failure mode:

  • handleMergeConflict() — a conflict on a non-ASCII-named file gets the escaped name, which fails the .md doc-file check (the escaped form has a trailing "), misroutes the file into the non-content auto-resolve path, and the git checkout --theirs -- <escaped> there fails → merge aborted, sync paused, instead of surfacing the conflict.
  • Conflict-store reconciliation (sync-engine.ts, boot.ts, conflict-storage.ts) — escaped names stored/compared, and resolve strategies reuse them as git rm/git add pathspecs.
  • dirtyFilesOverlapWith() (git-dirty.ts) — escaped names in the user-facing overlap list.
  • restoreSkillVersion() (skill-restore.ts) — escaped ls-tree output fed to git show, so skills containing non-ASCII-named files can't be restored.
  • diff-tree output used for auto-save commit messages — garbled names in the message.

Fix

All git path-listing invocations (ls-tree, diff-tree, diff --name-only, diff-index, status --porcelain) now pass -z and split on NUL via a shared splitNulSeparatedPaths helper in git-handle.ts, so paths round-trip as raw UTF-8 with no quoting to undo. The git-dirty.ts porcelain parser was updated for the -z record format (a rename's original path arrives as the following NUL record).

These sites change together deliberately: conflict paths written by the sync engine are consumed by conflict-storage.ts and boot.ts, so fixing only the writer would make real-vs-escaped comparisons wrongly clear live conflict entries.

Note for existing installs: conflicts.json entries persisted by older versions in escaped form are pruned at the next reconcile, and the existing stale-MERGE_HEAD handling recovers the merge state from git itself.

Tests

Written first and confirmed failing before the fix, passing after (their ASCII twins pass on main):

  • sync-engine.test.ts: deleting a committed non-ASCII-named file commits and pushes, with the real filename in the commit message; a remote-modify/local-delete conflict on a non-ASCII-named file surfaces as a content conflict with the real UTF-8 path.
  • git-dirty.test.ts: non-ASCII overlap reported with its real name; staged rename reports the new path (pins the -z porcelain record format).

bun run lint and bun run typecheck pass. The server suite passes apart from three failures that are unrelated and reproduce identically on a pristine main checkout in my environment (build-skill-bundles _shared asset guard, desktop IPC ratchet marker, CLI bundled jsonc write).

A changeset is included. Also manually verified on a real knowledge base running this build: deleting a file with umlauts in its name now commits and syncs.

…licts resolve, and commit messages show the real filename.

Git C-quotes non-ASCII filenames in its default output
("hyv\303\244\303\244 y\303\266t\303\244.md"), and the server parsed
that escaped string as a literal path. Reused as a pathspec it matches
nothing, so deleting a file with e.g. Nordic umlauts in its name was
never committed — every sync cycle concluded there was nothing to do.
The same parsing fed conflict detection (misrouting content conflicts
into the non-content auto-resolve path, which then aborted the merge
and paused sync), conflict-store reconciliation, dirty-overlap checks
before branch switches, skill version restore, and auto-save commit
messages.

All git path-listing calls now pass -z and split on NUL via a shared
splitNulSeparatedPaths helper, so paths round-trip as raw UTF-8.
The git-dirty porcelain parser reads the -z record format, where a
rename's original path arrives as the following NUL record.
@CLAassistant

CLAassistant commented Jul 4, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown

Thanks for the contribution! Your PR could not be merged automatically: it overlaps other changes that aren't visible here, so a maintainer needs to reconcile it by hand.

No action is needed from you. Your PR is already based on the latest inkeep/open-knowledge main; the overlap is on our side, not something to fix from your branch. Your commit attribution is preserved as @hannut.

A maintainer will resolve it and land your change; this PR will close automatically once it merges. This comment will be updated as the status changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants