Skip to content

Windows (zh-CN, ACP 936): terminal-selection annotations save CJK text as mojibake #76

Closed

Description

@Ody123zhou

Environment

  • herdr 0.9.3 (Windows 11 x86_64, system UI culture zh-CN)
  • plugin: plannotator/herdr-annotate/windows-full @ c5e565e
  • console code page: 936 (default for zh-CN). No system-wide UTF-8 beta enabled.
  • shell profile does not touch encoding; LANG/PYTHONIOENCODING irrelevant here.

What happens

  1. Select Chinese text in a pane, prefix+a, type a Chinese comment, Ctrl+S.
  2. The annotation is saved (good — it lands in %LOCALAPPDATA%\herdr\plugins\annotate\annotations.jsonl).
  3. But the stored text is mojibake: the selected text and the comment are both decoded with the ANSI code page instead of UTF-8.

Real record from my machine (one line of annotations.jsonl):

{"selectedText":"# Annotations on terminal selections\r\n\r\n## Annotation 1 (Code / 1)\r\nComment on:\r\n```\r\n# Annotations on terminal selections\r\n\r\n## Annotation 1 (Code / 1)\r\nComment on: \"������\"\r\n> 测试\r\n...

测试 (the comment I typed, in the popover) survived — it was typed through the popover input. The ������ is the selected terminal text, which came from herdr as ACP-encoded bytes and was treated as UTF-8 (or the reverse), so it round-trips as replacement characters.

So the split is:

  • text typed into the popover: preserved
  • text captured from the terminal selection: mangled for any non-ASCII content

Notes that may narrow it down

  • Document review path (plannotator-tui in the doc entrypoint) renders CJK correctly — I opened it with herdr plugin action invoke open --plugin annotate and the Chinese table in AGENTS.md displayed fine. So the TUI is OK; the problem looks specific to herdr-annotate.exe capture / the selection text path.
  • The doc-review pane was launched by me from a CLI process with a clean environment. When the same pane is opened through the keybinding from the herdr server, the user reported Chinese not rendering. Environment difference between those two launch paths may be worth a look (the server inherits a different PSModulePath on this machine, FWIW).
  • Related to the earlier fix "keep non-ascii text intact through the windows clipboard" — this looks like the same class of issue, on the selection-in path rather than the clipboard-out path.

Expected

Chinese (and any non-ASCII) selected text should be preserved byte-for-byte in the saved annotation, and should render correctly when copied back out as feedback.

Ask

Is the intended behavior to treat herdr's selection payload as UTF-8 unconditionally on Windows, or should the plugin read the console code page? Happy to test a patch if you point at the right spot.

Activity

  1. backnotprop commented on Oct 23, 2026

    @backnotprop
    Contributor

    Thanks @Ody123zhou. The fix from #74 now ships in helper 0.2.1 / plugin 0.9.2, so update the plugin and selections keep Chinese text.
    Please reopen if it still shows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions