Skip to content

fix(tokenizer): skip blank lines when loading a seg_dict file - #3903

Merged
LauraGPT merged 1 commit into
modelscope:mainfrom
MohammadHijjawi97:fix/char-tokenizer-seg-dict-blank-lines
Nov 15, 2026
Merged

LauraGPT merged 1 commit into
modelscope:mainfrom
MohammadHijjawi97:fix/char-tokenizer-seg-dict-blank-lines

fix(tokenizer): skip blank lines when loading a seg_dict file

bf61bd7
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

1 warning and 1 notice
numpy-compatibility (1.26.4)
succeeded Nov 14, 2026 in 3m 28s