Skip to content

Pull requests: huggingface/tokenizers

New pull request New
116 Open 910 Closed
116 Open 910 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

perf: reduce BPE tokenizer load-time memory
#2197 opened Jul 14, 2026 by xrl1 Loading…
Upgrade hf-hub to 1.0
#2196 opened Jul 14, 2026 by assafvayner Loading…
1
Clarify byte-level pretokenizer behavior
#2195 opened Jul 14, 2026 by michaelalbada Loading…
(wip) faster bpe run-pipeline-bench
#2194 opened Jul 13, 2026 by SBrandeis Contributor • Draft
[POC] fast encode: FlatCache + MPHF RankStore + incremental merge
#2190 opened Jul 12, 2026 by ArthurZucker Collaborator • Draft
1
fix: anchor ptrhash revision
#2189 opened Jul 11, 2026 by SBrandeis Contributor • Draft
4
feat: Use BucketVocabStore in PipelineBPE
#2188 opened Jul 11, 2026 by SBrandeis Contributor Loading…
Emit UserWarning from Split.pre_tokenize_str() on zero matches
#2186 opened Jul 10, 2026 by RUSHI-KOLLA Loading…
1
tk-encode: wire post-processor as pipeline STAGE_POST
#2182 opened Jul 10, 2026 by ArthurZucker Collaborator Loading…
3
perf(normalizer): SIMD-classified tag-driven normalization (1.3–8.6× across scripts) run-pipeline-bench
#2181 opened Jul 10, 2026 by ArthurZucker Collaborator Loading…
1
Fix PreTokenizedInputSequence annotation and tighten encode() stubs
#2180 opened Jul 10, 2026 by sridhar-3009 Loading…
1
decode: skip added-vocab check via id-range short-circuit
#2178 opened Jul 9, 2026 by sebpop Contributor Loading…
1
no alloc model, follow-ups
#2175 opened Jul 9, 2026 by SBrandeis Contributor Loading…
1
feat: T5 support (ai-generated)
#2174 opened Jul 9, 2026 by SBrandeis Contributor • Draft
1
chore(deps-dev): bump ws from 8.18.2 to 8.21.0 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2172 opened Jul 9, 2026 by dependabot Bot Loading…
1
dirty: thread-cache
#2170 opened Jul 8, 2026 by SBrandeis Contributor • Draft
1
Improve single thread performance of de-tokenization run-pipeline-bench
#2166 opened Jul 8, 2026 by sebpop Contributor Loading…
10
Clarifications for contributing guide
#2162 opened Jul 7, 2026 by connor-makowski Loading…
2
Remove unreachable convert_offsets empty 0..0 branch
#2159 opened Jul 6, 2026 by yaohwang Loading…
3 tasks done
Make doc builds faster
#2158 opened Jul 6, 2026 by mishig25 Contributor Loading…
1
Harden tokenizer deserialization against panics on crafted configs
#2151 opened Jul 5, 2026 by v-code01 Loading…
2
Add a atomsplit module run-pipeline-bench
#2150 opened Jul 5, 2026 by ArthurZucker Collaborator Loading…
1
Map prepended text to a zero-width span to avoid overlapping offsets (#1775)
#2149 opened Jul 5, 2026 by v-code01 Loading…
2
experiment: reduce normalizer alloc further
#2148 opened Jul 3, 2026 by SBrandeis Contributor • Draft
1
chore(deps): bump js-yaml from 3.14.2 to 3.15.0 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2131 opened Jun 30, 2026 by dependabot Bot Loading…
1
ProTip! Updated in the last three days: updated:>2026-07-11.