diff --git a/web-pages/product-site/content/legacy-manifest.json b/web-pages/product-site/content/legacy-manifest.json index c98d8395f..eb4a20b8d 100644 --- a/web-pages/product-site/content/legacy-manifest.json +++ b/web-pages/product-site/content/legacy-manifest.json @@ -144,6 +144,8 @@ "blog/websocket-offline-model-options.html": "9be8d17aaf2b99829442c0294568e00b49b9328b1480eba1d3d525cb65d7c1b1", "en/blog/websocket-offline-model-options.html": "bfd261068359bc529ae15e7830d518d26ecf4887d59312aad0f2e88b88805139", "blog/wav-pcm-input-checklist.html": "178ea5555257b9d3e0e7e4078f6616c0e2faf836fdd4bbfb269d934a5ffaa436", - "en/blog/wav-pcm-input-checklist.html": "3df930106634f89309db79c42abc352eace9c319556baac207938850d7cb8973" + "en/blog/wav-pcm-input-checklist.html": "3df930106634f89309db79c42abc352eace9c319556baac207938850d7cb8973", + "blog/dictionary-input-checklist.html": "d0ffbbc7e1464c592d07be173681c114d7f5d07d860a0ae41ba4cb223792090a", + "en/blog/dictionary-input-checklist.html": "896f32d930eb9630d1b8986a20afd139123e90debab423acc532cddc1dfb1b47" } } diff --git a/web-pages/product-site/data/blog.json b/web-pages/product-site/data/blog.json index 7cc95ac6e..f37ad07e5 100644 --- a/web-pages/product-site/data/blog.json +++ b/web-pages/product-site/data/blog.json @@ -9,6 +9,19 @@ ], "lead_image": "/img/funclip-v2-1-0-interface.jpg", "articles": [ + { + "slug": "dictionary-input-checklist", + "category": "explanations", + "reviewed": true, + "zh": { + "title": "分词词典加载失败?先检查空行与重复键", + "summary": "运行只读 UTF-8 词典检查器,定位空行、重复键、缺失映射和隐藏字符,明确主线修复与已发布安装包的边界。" + }, + "en": { + "title": "Dictionary loading failed? Check blank lines and duplicate keys", + "summary": "Run a read-only UTF-8 dictionary inspector to locate blank lines, duplicate keys, missing mappings and hidden characters, with explicit source-versus-package boundaries." + } + }, { "slug": "wav-pcm-input-checklist", "category": "explanations", diff --git a/web-pages/product-site/legacy/blog/dictionary-input-checklist.html b/web-pages/product-site/legacy/blog/dictionary-input-checklist.html new file mode 100644 index 000000000..1c262d597 --- /dev/null +++ b/web-pages/product-site/legacy/blog/dictionary-input-checklist.html @@ -0,0 +1,118 @@ + + + +分词词典加载失败?先检查空行与重复键 | FunASR + + + + + + + +

分词词典加载失败?先检查空行与重复键

+ +

准备好自己的词典,程序却在模型推理前就报错?先别调整模型参数。对使用 CharTokenizer 分词词典的流程,一行空白、重复键或文件开头的隐藏字符,就足以让加载或映射结果偏离预期。把文件结构检查从识别任务中拆出来,问题更容易复现,也不必先启动 GPU。

+

例如,hello h e l l o 表示键 hello 对应后面的分词单元。若文件后面再出现 hello hh,当前加载实现会由后一条覆盖前一条;只有一个 hello 而没有后续单元,则得到空映射。这些情况与“识别器是否认识 hello”不是同一个问题。

+

本文针对 FunASR 的 CharTokenizer 及其 seg_dict/seg_dict_file 路径,不是所有模型通用的词表或热词接口。检查器只读取本地 UTF-8 文本,使用 Python 标准库,不导入 FunASR、不加载权重、不联网、不改文件。示例数据是人工构造的文本,不构成语音准确率、训练效果或生产服务验收。

+

1. 先确认文件确实是这类词典

+

在固定提交的 CharTokenizer 实现中,加载函数按空白拆分每行:第一项是键,其余项连接成映射。制表符和多个空格可以作为分隔符。这里没有注释语法:以 # 开头的非空行仍会被当成词典条目。

+

分词路径会将待匹配词转为小写,但加载函数不会自动把词典键转为小写。因此,大写键值得核对。不要直接批量转换整个文件:输出单元还需要与目标模型的 token 表一致,盲目改写可能制造另一种错配。本文示例也不是可以直接替换任意模型词典的推荐词表。

+

2. 用只读检查器定位可疑行

+

把下面示例作为独立的 inspect_dictionary.py 运行,传入有权检查的词典文件。报告保留 1 起始行号;它记录空白行、无映射的键、重复键、键中的 BOM 字符及含大写字符的键。UTF-8 解码失败会直接报错,不用替换字符掩盖损坏。

+
import json
+import sys
+
+
+def inspect_seg_dict(path):
+    report = {
+        "entries": 0,
+        "blank_lines": [],
+        "empty_values": [],
+        "duplicate_keys": [],
+        "bom_lines": [],
+        "uppercase_keys": [],
+    }
+    seen = {}
+    with open(path, "r", encoding="utf-8") as source:
+        for number, line in enumerate(source, 1):
+            fields = line.strip().split()
+            if not fields:
+                report["blank_lines"].append(number)
+                continue
+            key = fields[0]
+            if len(fields) == 1:
+                report["empty_values"].append(number)
+            if "\ufeff" in key:
+                report["bom_lines"].append(number)
+            if key != key.lower():
+                report["uppercase_keys"].append(number)
+            if key in seen:
+                report["duplicate_keys"].append({
+                    "key": key,
+                    "previous_line": seen[key],
+                    "line": number,
+                })
+            seen[key] = number
+    report["entries"] = len(seen)
+    return report
+
+
+if __name__ == "__main__":
+    print(json.dumps(inspect_seg_dict(sys.argv[1]),
+                     ensure_ascii=False, indent=2))
+
python inspect_dictionary.py /path/to/seg_dict.txt
+

entries 是按原始键计算的唯一条目数,不是模型词表大小。previous_line 指最近一次同键出现的行号:三次重复会分别报告第二次和第三次覆盖。空文件返回 0 条;“没有发现可疑行”不代表词典可用。检查器不会验证输出单元是否在模型词表中,也不会模拟完整的未知词回退。

+

3. 按问题类型处理,而不是直接重写

+ +

4. 主线修复与安装版本要分开核对

+

#3905 的词典空行修复集成保留了原作者 #3903 的实现与测试,并把回归测试接入现有 CI。修复对空白拆分结果先做检查,再取首项,避免空行造成索引错误。相关 CPU 回归、原生 CUDA CTC 输入链路和两套 NumPy CI 已按实际测试范围验收;这不是预训练 ASR 准确率评测。

+

截至 2026-11-15 本次维护验收,修复已在 main,已验收的根包仍是 FunASR 1.4.26,不能宣称旧 wheel 已包含这项新修复。本文检查器不依赖该修复。升级前应核对目标发行说明、实际安装版本和最小复现,避免仅凭“仓库已合并”判断本机行为。

+

随文测试实际执行中英文源码页和构建页中的代码,覆盖空白与 CRLF、制表符、中文键、重复覆盖、空映射、BOM、大写键、空文件、非法 UTF-8、普通 # 条目和命令行入口,并确认输入文件字节不变。它不是对任意恶意或超大文件的安全审计。

+

词典结构通过后,再在隔离环境验证 token 映射及有权使用的真实样本。报告问题时提供脱敏的最小词典、版本和错误栈;词典可能包含内部实体或术语,不要直接公开完整文件。

+

若你要调整的是识别请求里的热词,而非分词映射,请继续阅读热词传递链路与安装包边界。

+
+ + + diff --git a/web-pages/product-site/legacy/en/blog/dictionary-input-checklist.html b/web-pages/product-site/legacy/en/blog/dictionary-input-checklist.html new file mode 100644 index 000000000..b96627f46 --- /dev/null +++ b/web-pages/product-site/legacy/en/blog/dictionary-input-checklist.html @@ -0,0 +1,118 @@ + + + +Dictionary loading failed? Check blank lines and duplicate keys | FunASR + + + + + + + +

Dictionary loading failed? Check blank lines and duplicate keys

+ +

Does your custom dictionary fail before model inference even starts? Check the file before changing model parameters. In a pipeline that uses a CharTokenizer segmentation dictionary, a blank line, a duplicate key or a hidden character can change loading or mapping behavior. A separate file inspection makes the problem reproducible without starting a GPU workload.

+

For example, hello h e l l o maps the key hello to the units that follow it. A later hello hh overrides that mapping in the current loader. A line containing only hello creates an empty mapping. None of these file-level observations tells you whether a recognizer can correctly recognize the spoken word.

+

This article covers FunASR's CharTokenizer and its seg_dict/seg_dict_file path. It is not a universal vocabulary or hotword API for every model. The inspector reads local UTF-8 text using the Python standard library. It does not import FunASR, load weights, access the network or modify files. The examples are constructed text fixtures, not speech-accuracy, training-quality or production-service evaluations.

+

1. Confirm the dictionary format first

+

In the pinned CharTokenizer implementation, the loader splits each line on whitespace: the first item is the key and the remaining items form its mapping. Tabs and repeated spaces can separate fields. There is no comment syntax here: a nonempty line beginning with # is still a dictionary entry.

+

The tokenization path lowercases input words before lookup, while the loader does not automatically lowercase dictionary keys. Uppercase keys therefore deserve inspection. Do not blindly normalize the whole file: mapped units must still match the target model's token vocabulary. The tiny example in this article is not a replacement vocabulary for arbitrary models.

+

2. Locate suspicious lines without changing the file

+

Run the following as an independent inspect_dictionary.py script with a dictionary you are authorized to inspect. It reports one-based line numbers for blank lines, keys without mapped units, duplicate keys, BOM characters inside keys and keys containing uppercase characters. Invalid UTF-8 raises a decoding error instead of silently replacing damaged bytes.

+
import json
+import sys
+
+
+def inspect_seg_dict(path):
+    report = {
+        "entries": 0,
+        "blank_lines": [],
+        "empty_values": [],
+        "duplicate_keys": [],
+        "bom_lines": [],
+        "uppercase_keys": [],
+    }
+    seen = {}
+    with open(path, "r", encoding="utf-8") as source:
+        for number, line in enumerate(source, 1):
+            fields = line.strip().split()
+            if not fields:
+                report["blank_lines"].append(number)
+                continue
+            key = fields[0]
+            if len(fields) == 1:
+                report["empty_values"].append(number)
+            if "\ufeff" in key:
+                report["bom_lines"].append(number)
+            if key != key.lower():
+                report["uppercase_keys"].append(number)
+            if key in seen:
+                report["duplicate_keys"].append({
+                    "key": key,
+                    "previous_line": seen[key],
+                    "line": number,
+                })
+            seen[key] = number
+    report["entries"] = len(seen)
+    return report
+
+
+if __name__ == "__main__":
+    print(json.dumps(inspect_seg_dict(sys.argv[1]),
+                     ensure_ascii=False, indent=2))
+
python inspect_dictionary.py /path/to/seg_dict.txt
+

entries counts unique original keys; it is not the model's vocabulary size. previous_line identifies the most recent earlier occurrence of a duplicate, so three occurrences produce two overwrite reports. An empty file reports zero entries. An empty warning list is not proof of a usable dictionary. The inspector neither checks output units against the model vocabulary nor simulates the full unknown-word fallback.

+

3. Resolve the cause instead of rewriting everything

+ +

4. Separate the merged fix from the installed package

+

The #3905 blank-line integration preserves the original #3903 implementation and test, and adds the regression to the existing CI entry point. It checks for an empty split result before accessing the first item. Relevant CPU regressions, a native CUDA CTC input chain and both NumPy CI configurations were accepted within their stated scope. This is not a pretrained ASR accuracy evaluation.

+

At the November 15, 2026 maintenance acceptance, the fix was on main while the accepted root package was still FunASR 1.4.26. The existing wheel must not be described as containing this new fix. The inspector above does not depend on the fix. Before upgrading, check the target release notes, actual installed version and a minimal reproduction instead of assuming that a merged PR describes your local installation.

+

The accompanying tests execute the code in both language sources and built pages. They cover whitespace and CRLF, tabs, Chinese keys, repeated overrides, empty mappings, BOM, uppercase keys, empty files, invalid UTF-8, ordinary # entries and the command-line entry point, and verify that input bytes remain unchanged. This is not a security audit of arbitrary malicious or oversized files.

+

After structural checks, validate the token mapping and authorized real samples in an isolated environment. A useful bug report includes a redacted minimal dictionary, versions and the error stack. Dictionaries may contain private names or terminology; do not publish the full file by default.

+

If you meant recognition hotwords rather than segmentation mappings, continue with hotword request paths and package boundaries.

+
+ + + diff --git a/web-pages/product-site/tests/test_seg_dict_article.py b/web-pages/product-site/tests/test_seg_dict_article.py new file mode 100644 index 000000000..363532fac --- /dev/null +++ b/web-pages/product-site/tests/test_seg_dict_article.py @@ -0,0 +1,111 @@ +"""Execute each published dictionary inspector against real UTF-8 files.""" +import contextlib +import io +import json +from pathlib import Path +import sys + +from bs4 import BeautifulSoup +import pytest + +SITE = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(SITE)) +from build import build + +SLUG = 'dictionary-input-checklist' +SOURCE = 'https://github.com/modelscope/FunASR/blob/d8a48a8af181dddf9b9f30aa50801c0ed6a6ffce/funasr/tokenizer/char_tokenizer.py' +FIELDS = ('blank_lines', 'empty_values', 'duplicate_keys', 'bom_lines', 'uppercase_keys') + +def expected(entries=0, **changes): + return dict(entries=entries, **{name: changes.get(name, []) for name in FIELDS}) + +def test_catalogue_discovers_dictionary_article_once(): + data = json.loads((SITE/'data/blog.json').read_text()) + rows = [x for x in data['articles'] if x['slug'] == SLUG] + assert len(rows) == 1 + assert rows[0]['reviewed'] and rows[0]['category'] == 'explanations' + assert SLUG not in data['selected'] + +@pytest.fixture(scope='module', params=['source', 'built']) +def tree(request, tmp_path_factory): + if request.param == 'source': + return SITE/'legacy' + dest = tmp_path_factory.mktemp('dictionary-article') + build(dest) + return dest + +@pytest.fixture(params=['', 'en/']) +def article(request, tree): + prefix = request.param + return prefix, BeautifulSoup((tree/prefix/'blog'/(SLUG+'.html')).read_text(), 'html.parser') + +@pytest.fixture +def inspector(article): + namespace = {'__name__': 'article_example'} + code = article[1].select_one('code[data-language="python"]').get_text() + exec(compile(code, 'article-dictionary-inspector', 'exec'), namespace) + return namespace['inspect_seg_dict'] + +def test_metadata_and_source_links(article): + prefix, soup = article + meta = json.loads(soup.select_one('script[type="application/ld+json"]').get_text()) + assert meta['datePublished'] == meta['dateModified'] == '2026-11-16' + assert meta['headline'] == soup.select_one('article h1').get_text() + assert soup.select_one('link[rel="canonical"]')['href'] == f'https://www.funasr.com/{prefix}blog/{SLUG}.html' + links = {a['href'] for a in soup.select('article a[href]')} + assert SOURCE in links and 'https://github.com/modelscope/FunASR/pull/3905' in links + assert 'https://github.com/modelscope/FunASR/releases/tag/v1.4.26' in links + assert soup.select_one('[data-editorial="boundary"]') + assert len(soup.select('[data-editorial="next-step"] a')) == 1 + +@pytest.mark.parametrize('text,count', [ + ('hello h e l l o\nworld w o r l d\n', 2), + ('\u4f60\u597d\t\u4f60\t\u597d\n', 1), + ('# this is a dictionary entry\n', 1), +]) +def test_clean_dictionary_and_no_implicit_comments(inspector, tmp_path, text, count): + p = tmp_path/'dictionary.txt'; p.write_text(text, encoding='utf-8') + before = p.read_bytes() + assert inspector(p) == expected(count) + assert p.read_bytes() == before + +@pytest.mark.parametrize('text,want', [ + ('hello h\n\n \t \nworld w\n', expected(2, blank_lines=[2, 3])), + ('hello h\r\n \t\r\nworld w\r\n', expected(2, blank_lines=[2])), + ('hello h\nworld w\nhello hh\nhello hhh\n', expected(2, duplicate_keys=[ + dict(key='hello', previous_line=1, line=3), dict(key='hello', previous_line=3, line=4)])), + ('hello\n', expected(1, empty_values=[1])), + ('\ufeffhello h\n', expected(1, bom_lines=[1])), + ('HELLO h\nHello h\n', expected(2, uppercase_keys=[1, 2])), +]) +def test_reports_exact_line_numbers_without_rewriting(inspector, tmp_path, text, want): + p = tmp_path/'dictionary.txt'; p.write_bytes(text.encode('utf-8')) + before = p.read_bytes() + assert inspector(p) == want + assert p.read_bytes() == before + +def test_empty_dictionary_is_not_an_invented_entry(inspector, tmp_path): + p = tmp_path/'empty.txt'; p.write_bytes(b'') + assert inspector(p) == expected() + assert p.read_bytes() == b'' + +def test_invalid_utf8_is_not_silently_replaced(inspector, tmp_path): + p = tmp_path/'invalid.txt'; p.write_bytes(b'hello \xff\n') + with pytest.raises(UnicodeDecodeError): + inspector(p) + assert p.read_bytes() == b'hello \xff\n' + +def test_cli_runs_actual_published_code(article, tmp_path, monkeypatch): + p = tmp_path/'input.txt'; p.write_text('hello h e l l o\n', encoding='utf-8') + monkeypatch.setattr(sys, 'argv', ['inspect_dictionary.py', str(p)]) + with contextlib.redirect_stdout(io.StringIO()) as out: + exec(compile(article[1].select_one('code[data-language="python"]').get_text(), 'article-cli', 'exec'), {'__name__': '__main__'}) + assert json.loads(out.getvalue()) == expected(1) + +@pytest.mark.parametrize('prefix', ['', 'en/']) +def test_archive_and_category_link_to_article(prefix, tmp_path): + build(tmp_path) + for category in ['archive', 'explanations']: + soup = BeautifulSoup((tmp_path/prefix/'blog'/category/'index.html').read_text(), 'html.parser') + cards = soup.select(f'a[href="/{prefix}blog/{SLUG}.html"]') + assert len(cards) == 1 and cards[0].select_one('time[datetime="2026-11-16"]')