Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion web-pages/product-site/content/legacy-manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,8 @@
"blog/websocket-offline-model-options.html": "9be8d17aaf2b99829442c0294568e00b49b9328b1480eba1d3d525cb65d7c1b1",
"en/blog/websocket-offline-model-options.html": "bfd261068359bc529ae15e7830d518d26ecf4887d59312aad0f2e88b88805139",
"blog/wav-pcm-input-checklist.html": "178ea5555257b9d3e0e7e4078f6616c0e2faf836fdd4bbfb269d934a5ffaa436",
"en/blog/wav-pcm-input-checklist.html": "3df930106634f89309db79c42abc352eace9c319556baac207938850d7cb8973"
"en/blog/wav-pcm-input-checklist.html": "3df930106634f89309db79c42abc352eace9c319556baac207938850d7cb8973",
"blog/dictionary-input-checklist.html": "d0ffbbc7e1464c592d07be173681c114d7f5d07d860a0ae41ba4cb223792090a",
"en/blog/dictionary-input-checklist.html": "896f32d930eb9630d1b8986a20afd139123e90debab423acc532cddc1dfb1b47"
}
}
13 changes: 13 additions & 0 deletions web-pages/product-site/data/blog.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,19 @@
],
"lead_image": "/img/funclip-v2-1-0-interface.jpg",
"articles": [
{
"slug": "dictionary-input-checklist",
"category": "explanations",
"reviewed": true,
"zh": {
"title": "分词词典加载失败?先检查空行与重复键",
"summary": "运行只读 UTF-8 词典检查器,定位空行、重复键、缺失映射和隐藏字符,明确主线修复与已发布安装包的边界。"
},
"en": {
"title": "Dictionary loading failed? Check blank lines and duplicate keys",
"summary": "Run a read-only UTF-8 dictionary inspector to locate blank lines, duplicate keys, missing mappings and hidden characters, with explicit source-versus-package boundaries."
}
},
{
"slug": "wav-pcm-input-checklist",
"category": "explanations",
Expand Down
118 changes: 118 additions & 0 deletions web-pages/product-site/legacy/blog/dictionary-input-checklist.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
<!DOCTYPE html>

<html lang="zh"><head><meta charset="utf-8"/><meta content="width=device-width,initial-scale=1" name="viewport"/>
<title>分词词典加载失败?先检查空行与重复键 | FunASR</title><meta content="运行只读 UTF-8 词典检查器,定位空行、重复键、缺失映射和隐藏字符,明确主线修复与已发布安装包的边界。" name="description"/>
<link href="/logo.png" rel="icon"/><meta content="分词词典加载失败?先检查空行与重复键" property="og:title"/><meta content="运行只读 UTF-8 词典检查器,定位空行、重复键、缺失映射和隐藏字符,明确主线修复与已发布安装包的边界。" property="og:description"/>
<meta content="article" property="og:type"/><meta content="https://www.funasr.com/blog/dictionary-input-checklist.html" property="og:url"/><meta content="https://www.funasr.com/logo.png" property="og:image"/>
<link href="https://www.funasr.com/blog/dictionary-input-checklist.html" rel="canonical"/><link href="https://www.funasr.com/en/blog/dictionary-input-checklist.html" hreflang="en" rel="alternate"/><link href="https://www.funasr.com/blog/dictionary-input-checklist.html" hreflang="zh" rel="alternate"/><link href="https://www.funasr.com/en/blog/dictionary-input-checklist.html" hreflang="x-default" rel="alternate"/>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&amp;display=swap" rel="stylesheet"/>
<script type="application/ld+json">{"@context":"https://schema.org","@type":"BlogPosting","headline":"分词词典加载失败?先检查空行与重复键","description":"运行只读 UTF-8 词典检查器,定位空行、重复键、缺失映射和隐藏字符,明确主线修复与已发布安装包的边界。","datePublished":"2026-11-16","dateModified":"2026-11-16","author":{"@type":"Organization","name":"FunASR / Tongyi Lab"},"publisher":{"@type":"Organization","name":"FunASR","logo":{"@type":"ImageObject","url":"https://www.funasr.com/logo.png"}},"mainEntityOfPage":{"@type":"WebPage","@id":"https://www.funasr.com/blog/dictionary-input-checklist.html"},"image":"https://www.funasr.com/logo.png"}</script>
<style>:root{--primary:#2563eb;--primary-dark:#1d4ed8;--bg:#fff;--surface:#f8fafc;--border:#e2e8f0;--text:#0f172a;--text-soft:#475569;--text-muted:#94a3b8;--radius:12px}
*{margin:0;padding:0;box-sizing:border-box}
body{font-family:'Inter',-apple-system,sans-serif;color:var(--text);line-height:1.7}
a{color:var(--primary);text-decoration:none}a:hover{text-decoration:underline}
.container{max-width:800px;margin:0 auto;padding:0 24px}
.nav{position:fixed;top:0;left:0;right:0;z-index:100;background:rgba(255,255,255,0.9);backdrop-filter:blur(12px);border-bottom:1px solid var(--border);padding:14px 0}
.nav .container{max-width:1120px;display:flex;align-items:center;gap:24px}
.nav-logo{font-size:1.2rem;font-weight:800;color:var(--text)}.nav-logo span{color:var(--primary)}
.nav-links{display:flex;gap:20px;margin-left:auto;flex-wrap:wrap}
.nav-links a{font-size:0.85rem;font-weight:500;color:var(--text-soft)}
.nav-btn{padding:7px 16px;background:var(--primary);color:#fff;border-radius:7px;font-size:0.83rem;font-weight:600}
article{padding:120px 0 80px}
h1{font-size:2rem;font-weight:800;line-height:1.3;margin-bottom:14px}
h2{font-size:1.3rem;font-weight:700;margin:36px 0 14px;padding-top:16px;border-top:1px solid var(--border)}
h3{font-size:1.05rem;font-weight:700;margin:22px 0 10px}
p{margin-bottom:14px;color:var(--text-soft)}
ul,ol{margin:0 0 14px 22px;color:var(--text-soft)}li{margin-bottom:6px}
table{width:100%;border-collapse:collapse;margin:16px 0;font-size:0.86rem}
th,td{padding:10px 12px;text-align:left;border-bottom:1px solid var(--border)}
th{font-weight:600;background:var(--surface);font-size:0.78rem;color:var(--text-muted)}
pre{background:#1e293b;color:#e2e8f0;padding:18px;border-radius:10px;overflow-x:auto;font-size:0.82rem;line-height:1.7;margin:14px 0}
code{font-family:ui-monospace,Menlo,monospace}
p code,li code{background:var(--surface);padding:2px 6px;border-radius:5px;font-size:0.85em;color:var(--primary-dark)}
.post-meta{color:var(--text-muted);font-size:0.85rem;margin-bottom:24px}
.cta{background:var(--surface);border:1px solid var(--border);border-radius:var(--radius);padding:24px;margin:28px 0;text-align:center}
.cta a.btn{display:inline-block;padding:10px 24px;background:var(--primary);color:#fff;border-radius:8px;font-weight:600}
.post-list{list-style:none;margin:0;padding:0}
.post-list li{margin:0 0 8px}
.post-card{display:block;border:1px solid var(--border);border-radius:var(--radius);padding:20px 22px;margin-bottom:14px}
.post-card:hover{border-color:var(--primary);text-decoration:none}
.post-card h2{border:none;padding:0;margin:0 0 6px;font-size:1.15rem;color:var(--text)}
.post-card p{margin:0;color:var(--text-soft);font-size:0.9rem}
.post-card .date{color:var(--text-muted);font-size:0.8rem}
footer{background:#0f172a;color:#94a3b8;padding:28px 0;text-align:center;font-size:0.8rem}
footer a{color:#94a3b8}
@media(max-width:900px){.nav-links{display:none}.nav .container{gap:16px}.nav-logo{margin-right:auto}h1{font-size:1.75rem}table{table-layout:fixed;font-size:0.78rem}th,td{padding:8px 4px;overflow-wrap:anywhere}p code,li code{overflow-wrap:anywhere;word-break:break-word}}
article p a,article li a{color:#006a57;text-decoration:underline;text-underline-offset:.15em}
</style></head><body>
<nav class="nav"><div class="container"><a class="nav-logo" href="/">Fun<span>ASR</span></a><div class="nav-links"><a href="/#capabilities">核心能力</a><a href="/#usage">如何使用</a><a href="/#benchmark">性能</a><a href="/#demo">演示</a><a href="https://modelscope.github.io/FunASR/zh/" target="_blank">文档</a><a href="/quickstart.html">快速上手</a><a href="/models.html">模型</a><a href="/ecosystem.html">生态</a><a href="/blog/">技术博客</a><a href="/vs-whisper.html">对比 Whisper</a><a href="/donors.html">功德榜</a></div><a href="https://www.funasr.com/en/blog/dictionary-input-checklist.html" style="font-size:0.83rem;font-weight:600;color:var(--text-soft)">EN</a><a class="nav-btn" href="/go/github" target="_blank">GitHub ★</a></div></nav>
<article><div class="container"><h1>分词词典加载失败?先检查空行与重复键</h1>
<p class="post-meta">FunASR Blog · 2026-11-16</p>
<p class="lead" data-editorial="opening">准备好自己的词典,程序却在模型推理前就报错?先别调整模型参数。对使用 CharTokenizer 分词词典的流程,一行空白、重复键或文件开头的隐藏字符,就足以让加载或映射结果偏离预期。把文件结构检查从识别任务中拆出来,问题更容易复现,也不必先启动 GPU。</p>
<p data-editorial="example">例如,<code>hello h e l l o</code> 表示键 hello 对应后面的分词单元。若文件后面再出现 <code>hello hh</code>,当前加载实现会由后一条覆盖前一条;只有一个 hello 而没有后续单元,则得到空映射。这些情况与“识别器是否认识 hello”不是同一个问题。</p>
<p data-editorial="boundary">本文针对 FunASR 的 <code>CharTokenizer</code> 及其 <code>seg_dict</code>/<code>seg_dict_file</code> 路径,不是所有模型通用的词表或热词接口。检查器只读取本地 UTF-8 文本,使用 Python 标准库,不导入 FunASR、不加载权重、不联网、不改文件。示例数据是人工构造的文本,不构成语音准确率、训练效果或生产服务验收。</p>
<h2>1. 先确认文件确实是这类词典</h2>
<p>在<a href="https://github.com/modelscope/FunASR/blob/d8a48a8af181dddf9b9f30aa50801c0ed6a6ffce/funasr/tokenizer/char_tokenizer.py">固定提交的 CharTokenizer 实现</a>中,加载函数按空白拆分每行:第一项是键,其余项连接成映射。制表符和多个空格可以作为分隔符。这里没有注释语法:以 # 开头的非空行仍会被当成词典条目。</p>
<p>分词路径会将待匹配词转为小写,但加载函数不会自动把词典键转为小写。因此,大写键值得核对。不要直接批量转换整个文件:输出单元还需要与目标模型的 token 表一致,盲目改写可能制造另一种错配。本文示例也不是可以直接替换任意模型词典的推荐词表。</p>
<h2>2. 用只读检查器定位可疑行</h2>
<p>把下面示例作为独立的 <code>inspect_dictionary.py</code> 运行,传入有权检查的词典文件。报告保留 1 起始行号;它记录空白行、无映射的键、重复键、键中的 BOM 字符及含大写字符的键。UTF-8 解码失败会直接报错,不用替换字符掩盖损坏。</p>
<pre><code data-language="python">import json
import sys


def inspect_seg_dict(path):
report = {
"entries": 0,
"blank_lines": [],
"empty_values": [],
"duplicate_keys": [],
"bom_lines": [],
"uppercase_keys": [],
}
seen = {}
with open(path, "r", encoding="utf-8") as source:
for number, line in enumerate(source, 1):
fields = line.strip().split()
if not fields:
report["blank_lines"].append(number)
continue
key = fields[0]
if len(fields) == 1:
report["empty_values"].append(number)
if "\ufeff" in key:
report["bom_lines"].append(number)
if key != key.lower():
report["uppercase_keys"].append(number)
if key in seen:
report["duplicate_keys"].append({
"key": key,
"previous_line": seen[key],
"line": number,
})
seen[key] = number
report["entries"] = len(seen)
return report


if __name__ == "__main__":
print(json.dumps(inspect_seg_dict(sys.argv[1]),
ensure_ascii=False, indent=2))</code></pre>
<pre><code data-language="shell">python inspect_dictionary.py /path/to/seg_dict.txt</code></pre>
<p><code>entries</code> 是按原始键计算的唯一条目数,不是模型词表大小。<code>previous_line</code> 指最近一次同键出现的行号:三次重复会分别报告第二次和第三次覆盖。空文件返回 0 条;“没有发现可疑行”不代表词典可用。检查器不会验证输出单元是否在模型词表中,也不会模拟完整的未知词回退。</p>
<h2>3. 按问题类型处理,而不是直接重写</h2>
<ul>
<li><strong>空白行:</strong>先确认安装代码是否包含对应修复;需要兼容旧环境时,人工审阅后在另一份文件中去除意外空行,保留原文件和差异。</li>
<li><strong>重复键或空映射:</strong>核对词典来源和生成流程,明确哪条才是期望映射。不要让“最后一条生效”悄悄替代业务规则。</li>
<li><strong>BOM 或编码错误:</strong>确认文件编码和导出工具,在保留原件的前提下做明确的编码转换;不要用忽略解码错误的方式继续加载。</li>
<li><strong>大写键:</strong>结合实际输入和模型 token 表检查大小写策略。结构检查不能确认输出单元的语义正确。</li>
</ul>
<h2>4. 主线修复与安装版本要分开核对</h2>
<p><a href="https://github.com/modelscope/FunASR/pull/3905">#3905 的词典空行修复集成</a>保留了原作者 #3903 的实现与测试,并把回归测试接入现有 CI。修复对空白拆分结果先做检查,再取首项,避免空行造成索引错误。相关 CPU 回归、原生 CUDA CTC 输入链路和两套 NumPy CI 已按实际测试范围验收;这不是预训练 ASR 准确率评测。</p>
<p>截至 2026-11-15 本次维护验收,修复已在 main,已验收的根包仍是 <a href="https://github.com/modelscope/FunASR/releases/tag/v1.4.26">FunASR 1.4.26</a>,不能宣称旧 wheel 已包含这项新修复。本文检查器不依赖该修复。升级前应核对目标发行说明、实际安装版本和最小复现,避免仅凭“仓库已合并”判断本机行为。</p>
<p>随文测试实际执行中英文源码页和构建页中的代码,覆盖空白与 CRLF、制表符、中文键、重复覆盖、空映射、BOM、大写键、空文件、非法 UTF-8、普通 # 条目和命令行入口,并确认输入文件字节不变。它不是对任意恶意或超大文件的安全审计。</p>
<p>词典结构通过后,再在隔离环境验证 token 映射及有权使用的真实样本。报告问题时提供脱敏的最小词典、版本和错误栈;词典可能包含内部实体或术语,不要直接公开完整文件。</p>
<p data-editorial="next-step">若你要调整的是识别请求里的热词,而非分词映射,请继续阅读<a href="/blog/hotwords-package-boundaries.html">热词传递链路与安装包边界</a>。</p>
</div></article>
<footer><div class="container">FunASR · 通义实验室开源语音识别 · <a href="/go/github">GitHub</a></div></footer>
</body></html>

Loading
Loading