Skip to content

Pull requests: huggingface/tokenizers

New pull request New
68 Open 835 Closed
68 Open 835 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Document that train_new_from_iterator uses BPE for WordPiece
#2065 opened May 21, 2026 by adityasingh2400 Loading…
3 tasks done
1
Replace dead wikitext s3 link in quicktour with HF dataset mirror
#2064 opened May 21, 2026 by adityasingh2400 Loading…
2
chore(deps): bump qs and body-parser in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2063 opened May 20, 2026 by dependabot Bot Loading…
1
chore(deps-dev): bump webpack-dev-server from 5.2.1 to 5.2.4 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2062 opened May 20, 2026 by dependabot Bot Loading…
1
Fix Encode Inputs doc page rendering (closes #1909)
#2061 opened May 18, 2026 by LeSingh1 Loading…
1
security: reject nested-quantifier regex in Split/Replace to prevent ReDoS (CWE-1333)
#2060 opened May 17, 2026 by Allen930311 Loading…
4 tasks
fix(bpe): widen pair_counts from i32 to i64 to prevent overflow on large corpora
#2059 opened May 17, 2026 by xodn348 Loading…
1
Fix i32 overflow in BPE trainer pair_counts on large corpora
#2057 opened May 14, 2026 by petyb • Draft
serialize tokenizer vocab and added_tokens compactly
#2056 opened May 13, 2026 by ArthurZucker Collaborator Loading…
1
Apply type_ids and sequence_id to overflow encodings in post-processors
#2055 opened May 12, 2026 by 1fanwang Loading…
1
Fix invalid escape sequence in Whitespace docstring
#2054 opened May 10, 2026 by eyupcanakman Loading…
1
Add scaling_bench: encode_batch vs worker-pool comparison (#1900)
#2048 opened May 1, 2026 by stargazerZJ Loading…
5 of 6 tasks
1
WIP: testing only
#2047 opened Apr 30, 2026 by assafvayner • Draft
4 tasks
1
Cyrillic normalizer and decoder for south slavic languages
#2046 opened Apr 28, 2026 by procesaur Loading…
Batch encode: simple lock-free scheduler
#2044 opened Apr 28, 2026 by sebpop Contributor Loading…
4
perf(unigram): pre-size token map and replace per-node HashMap with Vec
#2039 opened Apr 26, 2026 by taeyun16 Loading…
1
feat(ByteLevel): skip per-byte transform for printable-ASCII tokens
#2038 opened Apr 26, 2026 by KimYannn Loading…
2 of 3 tasks
3
feat(NFC): skip Unicode pass for all-ASCII inputs
#2037 opened Apr 26, 2026 by KimYannn Loading…
2 of 3 tasks
2
feat: SIMD ASCII fast path for Lowercase normalizer (~30-49x)
#2036 opened Apr 26, 2026 by KimYannn Loading…
6 of 7 tasks
2
V0.23 release
#2032 opened Apr 24, 2026 by ArthurZucker Collaborator Loading…
1
perf(byte_level): port GPT-2 split regex to logos FSM (−22% on GPT-2 encode)
#2031 opened Apr 23, 2026 by ArthurZucker Collaborator Loading…
2
Real-world (batch × input_length) tokenizer benchmark + cross-library leaderboard
#2030 opened Apr 23, 2026 by ArthurZucker Collaborator Loading…
4 tasks
1
perf: skip alignment tracking in encode_fast normalization
#2022 opened Apr 10, 2026 by ArthurZucker Collaborator Loading…
5
feat: Normalizer::normalize_str — skip NormalizedString allocation
#2020 opened Apr 10, 2026 by ArthurZucker Collaborator Loading…
1
Reduce crate size
#2015 opened Apr 9, 2026 by ArthurZucker Collaborator Loading…
6
ProTip! Type g i on any issue or pull request to go back to the issue listing page.