登录

tokenizers

huggingfaceApache-2.0活跃维护

用 Rust 实现的高性能分词器库，支持 BERT、GPT 等主流模型，兼顾研究实验与生产部署，解决文本预处理速度慢、跨语言调用不便的问题。

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

v0.23.3发布于 2026-10-09 10:20 UTC11,230star +3 / 7 天1,251fork同步于 2026-11-10 05:29 UTC
分词器文本预处理RustBERTGPT
对比 GitHub 官网bertgptlanguage-modelnatural-language-processingnatural-language-understanding

怎么上手

由 README 提炼，执行前请确认
原文 
pip install --pre tokenizers
硬件要求按 README 说明

README 未说明显存要求；不强制 SIMD，所有算子都有可移植的标量实现作为正确性基准，aarch64 用 NEON、x86_64 运行时检测 AVX-512 VBMI/SSE4.1、wasm32 需 simd128 目标特性，缺失时走标量路径。

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

Hugging Face 出品的工业级分词引擎，覆盖所有语言与模型，兼顾研究与生产

用 Rust 实现的高性能分词器库，支持 BERT、GPT 等主流模型，兼顾研究实验与生产部署，解决文本预处理速度慢、跨语言调用不便的问题。

核心能力

  • 用 compile() 生成 CompiledTokenizer 执行编解码
  • 内置 BPE、Unigram、WordPiece、WordLevel 模型
  • bitcannon 以位流程序实现预分词，自带多套字节级语法
  • 提供 Python、Node.js、WebAssembly 绑定，Rust 为参考实现
  • 训练与 Hub 下载为可选 feature，可按需裁剪二进制体积

容易踩的坑

当前为 v1.0.0.rc 预发布版，功能未全部就绪，需用 --pre 安装；Rust 侧 train 与 http 为默认关闭的可选 feature，需显式开启；README 提到 ModernBERT / OLMo 加载存在 64 位哈希碰撞问题待修复。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v0.23.3
全部 release
  1. v1.0.0-rc.4v1.0.0-rc.4 Release candidatepre-release2026-11-09 · 2026-11-09 14:23 UTC
    查看更新说明

    v1.0.0-rc.4

    Release Candidate number 4 is out! It brings back the required Python APIs to use tokenizers in other python libraries, for example transformers, vLLM, ...

    Install it with:

    pip install --pre tokenizers==1.0.0rc4

    We're making steady progress towards feature parity with 0.23 on encoding and decoding, the main milestone left before a stable 1.0.0!

    Breaking changes from v1.0.0-rc.3

    If you built something on a previous release candidate, here are the breaking changes from v1.0.0-rc.3:

    • Python: The Tokenizer class no longer encodes, you need to call tokenizer.compile() first. A Tokenizer now holds the tokenizer.json config and can be edited. compile() builds a frozen CompiledTokenizer from it. CompiledTokenizer can encode and decode. (#2562)

      tokenizer = Tokenizer.from_pretrained("openai-community/gpt2")
tokenizer.truncation = Truncation(max_length=512)
compiled = tokenizer.compile()
compiled.encode("Hello, World!").ids

    Benefits:

    • Encoding takes no lock and releases the GIL
    • Copying a Tokenizer is cheap: copies share the config until one of them is edited
    • Padding and truncation live in the config and become the encode defaults; each encode call can still override them

    New

    • Python: composition API. Build a tokenizer from components, or swap one out on an existing tokenizer:
      • tokenizer.normalizer, .pre_tokenizer, .model, .post_processor and .decoder can be read and set. Components are shared like in 0.23: editing one updates every tokenizer using it (#2579)
      • Models: BPE, Unigram, WordLevel, WordPiece, and Tokenizer(model) (#2569)
      • Post-processors: TemplateProcessing, RobertaProcessing, BertProcessing, ByteLevel and Sequence (#2573)
      • Sequence for normalizers (#2577)
      • Component classes are now generated from the Rust declarations, so adding the remaining ones is cheap (#2629)
    • Python: serialization. Tokenizer.from_str, to_str and save (#2563)
    • Python: vocab APIs. get_vocab, get_vocab_size, token_to_id and id_to_token (#2614)
    • Python: added tokens. AddedToken, Tokenizer.add_tokens, Tokenizer.add_special_tokens and CompiledTokenizer.added_tokens. Adding tokens one at a time is ~1000x faster than 0.23.1 (5k calls: 4 ms vs 3.85 s) (#2615)
    • Python: DecodeStream, compatible with vLLM's usage (#2551)

    Performance

    • Added tokens use ~1000x less memory: gpt-oss goes from 1.71 MiB to 1.6 KiB. Decode is up to 20% faster (#2612)
    • Many threads encoding at once no longer build a throwaway thread pool when the shared one is busy: up to 3x the throughput with 32 concurrent callers (#2633)
    • Loading a BPE model builds its vocabulary once instead of three times (#2613)

    Fixes

    • Fixes a bug causing forked child processes to hang when they loaded or unpickled a tokenizer after the parent had loaded one. This is the pattern behind datasets.map(num_proc=...) and PyTorch DataLoader fork workers (#2581, #2632)
    • Fixes a bug causing byte-level BPE tokenizers whose vocabulary lacks some bytes to fail to load. They load again with a warning, and encoding drops the missing bytes, like 0.23 (#2550)
    • Fixes a bug causing WordPiece tokenizers with a "\0" token in their vocabulary (e.g. ai4bharat/IndicBERTv2-MLM-only) to fail to load (#2661)
    • Fixes a bug causing Split pre-tokenizers read from tokenizer.json to ignore the fancy-regex backend when it is enabled (#2547)

    Internal

    • Dependency updates, CI and README maintenance (#2575, #2582, #2587, #2606, #2609, #2610, #2621, #2659)

    Contributors

    • @SBrandeis
    • @McPatate
    • @ArthurZucker
    • @dlowzzxx made their first contribution 🎉 Thank you!

    Full Changelog: https://github.com/huggingface/tokenizers/compare/v1.0.0-rc.3...v1.0.0-rc.4

    This changelog has been written with AI assistance

  2. v1.0.0-rc.3Release candidate v1.0.0-rc.3pre-release2026-10-19 · 2026-10-19 17:08 UTC
    查看更新说明

    v1.0.0-rc.3

    1.0.0-rc.3 is mainly a bindings release: it brings tokenizers to the browser with WebAssembly bindings!

    Install them with:

    npm install tokenizers-wasm@next

    We're making steady progress towards feature parity with 0.23, the main milestone left before a stable 1.0.0.

    New

    • WebAssembly bindings, published on npm as tokenizers-wasm. Release candidates go out under the next dist-tag (#2450, #2497, #2552)
    • Python: encode pairs of text (#2453)

    Fixes

    • Fixes a bug causing byte-level BPE tokenizers whose vocabulary lacks bytes that can never appear in UTF-8 (e.g. ModernBERT) to fail to load. The error now names the missing byte (#2466)
    • Fixes a bug causing WordPiece tokenizers with an empty-string token in their vocabulary to fail to load (#2538)
    • Fixes a bug causing the Python, Node and WASM bindings to reject any tokenizer.json with a regex Split pre-tokenizer or Replace normalizer, which covers NLLB and every model transformers converted from SentencePiece (#2536)
    • Fixes a bug causing data races when Python's Encoding.*_array getters were used concurrently; they now return an owned copy instead of a view (#2464)
    • Fixes a bug causing a 64-bit hash collision between two tokens to silently drop one of them when loading a vocabulary; it is now an error. Duplicate tokens log a warning instead of printing the whole vocab to stdout (#2467)
    • Fixes a bug causing ParityBpeTrainer to produce a different output than the regular BPE trainer; it now trains into Vocab + Merges like the others (#2474)

    Python

    • huggingface_hub 2.0 is allowed (#2486)
    • One unnecessary Encoding copy at the Rust/Python boundary is gone (#2452)

    Internal

    • Umbrella crate features: depend on tokenizers alone and pick components by feature, e.g. cargo add tokenizers -F train or cargo add tokenizers -F bpe --no-default-features. The tk-* crates still work on their own (#2476, #2477, #2479)
      • The bindings now depend on the umbrella crate (#2478, #2552)
      • The crate README and docs.rs front page now explain how to install the crate and pick its features (#2481)
    • Dependency updates, CI hardening and other maintenance (#2446, #2448, #2449, #2465, #2471, #2472, #2473, #2475, #2480, #2492, #2495, #2510, #2537, #2548)

    Thank you

    • @SBrandeis
    • @paulinebm
    • @musicinmybrain

    Full Changelog: https://github.com/huggingface/tokenizers/compare/v1.0.0-rc.2...v1.0.0-rc.3

    This changelog has been written with AI assistance

  3. v0.23.3Release v0.23.3 (python only)最新正式版2026-10-09 · 2026-10-09 10:20 UTC

    Release v0.23.3

    Python package release, other packages are unchanged

    :rotating_light: Fixs a breaking change introduced in v0.23.1

    What's changed

    Patch for the python package with three fixes:

    Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.23.2...v0.23.3

  4. v1.0.0-rc.2Release candidate v1.0.0.rc.2pre-release2026-09-21 · 2026-09-21 11:28 UTC
    查看更新说明

    Making tokenizers the sota tokenization library, same API, same token IDs, same standards

    We are happy to finally bring tokenizers up to speed :zap: ! tokenizers was released 7 years ago, in a time where running the model was always the bottleneck. The ecosystem has evolved so much that tokenizers became a the bottleneck.

    As we saw many different libraries and project pop out, improving the tokenizer, we realized that our library's entire stack was too outdated for external contributors to contribute. This pushed us to refactor the codebase, and make sure the next developers come here to push the performances of tokenization.

    The TLDR; is that we're a lot faster than before, crate size 6x smaller, peak memory is reduced. We worked a lot to reduce the UTF-8 tax, and make sure the improvements span across all languages.

    For more details on the release, read this.

    Big Kudos to @SBrandeis and @McPatate for this sprint. Special thanks to @sebpop for his involvement as well.

  5. v0.23.22026-09-03 · 2026-09-03 10:09 UTC
    查看更新说明

    This is the last v0 release, we are moving to v1!!

    More details coming soon 👀

    What's Changed

    New Contributors

    Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.23.1...v0.23.2

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    The SOTA tokenization library on all languages, all models, all hardwares.

    Our goal with tokenizers is to develop and maintain the industry's standard tokenization engine, making it the defacto place for everyone to contribute to the whole ecosystem.

    Release candidate: v1.0.0.rc.x

    As we are switching from 0.23 to v1.0.0, the current library does not ship all features. If you are worried, check the v1 features section for more details of what we are bringing back. We will publish a blog about what breaking changes we introduced. We strived to keep them as small as possible.

    Installation

    pip install --pre tokenizers

    For Rust, add the crate to your Cargo.toml:

    [dependencies]
tokenizers = "1.0.0"

    Training (train) and downloading from the Hub (http) are opt-in features. The crate's Using the crate section lists every feature and how to build a smaller binary.

    Usage

    >>> from tokenizers import Tokenizer
>>> tokenizer = Tokenizer.from_pretrained("meta-llama/Llama-3.1-8B")   # or .from_file(path)
>>> compiled = tokenizer.compile()
>>> compiled.encode("Hello, y'all! How are you 😁 ?")
Encoding(ids=[128000, 9906, 11, 379, 65948, 0, 2650, 527, 499, 27623, 223, 949], type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
>>> compiled.encode(("What is this?", "A pair.")).type_ids
[0, 0, 0, 0, 0, 1, 1, 1, 1]
>>> compiled.tokenize("Hello")
['<|begin_of_text|>', 'Hello']
>>> compiled.decode([15339, 1917])
'hello world'
>>> compiled.decode_tokens(compiled.encode("Hello"))
['<|begin_of_text|>', 'Hello']

    Tokenizer holds the tokenizer.json config and can be edited (e.g. tokenizer.truncation = ...); compile() builds the CompiledTokenizer that encodes and decodes.

    Performances

    Bindings

    status
    Rust✅ reference implementation
    Python✅
    Node.js✅
    WebAssembly✅ tokenizers-wasm on npm
    C / C++ / Java / Go/ etc🚧 planned
    Rubycommunity, external repo

    v1.0.0 roadmap:

    As we work toward v1, we are gonna bring back the entire python API that allows transformers' style of interacting with a tokenizer object:

    TODOS:

    Python Bindings

    • Bring training back: this should be easy, but we want the perfs gains to come as well.
    • Batch encoding API with zero-copy views on encoded ids
    • dlpack API for torch / JAX / numpy interop.
    • Composition: edit components of the pipeline (tokenizer.normalizer = [...]). The editable Tokenizer spec and compile() have landed; per-component handles are next.
    • tokenizer.save to persist a modified or trained tokenizer on disk.
    • role_to_token override.
    • Bring back offset output: this should be useful for people training.
    • Windowing / stride in encode.
    • decode_stream.

    Rust core

    • bitnorm: more performance from optimized normalization, as this will bring breaking changes we want to make sure it gets to v1.
    • Windowing (truncation with stride).
    • Improve training performance
    • Improve decode performance
    • Offset tracking
    • Padding: 2D flat buffer
    • Wordcache: shrn vs emulated movemask
    • Fix ModernBERT / OLMo load failure ("64-bit hash collision").
    • Unroll more regex: we need to cover the most used ones, we might have missed one or two.

    Other

    • C & C++ bindings for loading, encode and decode APIs

    After 1.0.0

    • Improve / rewrite the bitcannon crate
    • Java, Go, Swift... bindings, supported by the C bindings
    • tk-devices — exploratory GPU encoding and batch decoding that keeps text and ids on the device: upload the vocabulary once, compute output positions in parallel, gather the bytes on the GPU. An optional component aimed at large batches, subject to prototyping and measurement.

    Crate details

    Crate size

    Sub-crates

    The models (BPE, Unigram, WordPiece, WordLevel) and the full pipeline: Normalizer, PreTokenizer, Model, PostProcessor, Decoder.

    Why separate: you don't need all features to serve a model in production settings. Splitting it from training means a serving binary never links a trainer, a corpus reader, legacy converters, etc.

    Unicode tag classification plus pre-tokenization as a bitstream program. Follows Interleaved Bitstream Execution for Multi-Pattern Regex Matching on GPUs (MICRO'25, 10.1145/3725843.3756052) but we added our own algorithm to make sure we don't need a parabix engine. Characters are first classified into "tags", which use a sparse table generated when compiling bitcannon. This means there are 0 depencencies to the heavy unicode tables.

    Ships byte-exact grammars for gpt2 / ByteLevel, cl100k, o200k, tekken, deepseek and kimi-k2. The classes or tags each have a high and low nibble (encoded on a u8), and carry refinement when needed. Using u8 tags allows us to leverage less bitsream, meaning have a simpler / faster pre tokenization.

    Why separate: we just thought it might be useful to some people someday. As far as we know, this implementation is the fastest for every single pretokenizer regex on CPU.

    from_json_file turns a canonical tokenizer.json into a PipelineTokenizer.

    Why separate: this just makes it optional for some usecases!

    canonicalize_file rewrites legacy a tokenizer.json written by any older version of this library into the canonical form the reader accepts. A pure JSON→JSON rewrite; it depends on nothing but std::path and serde_json.

    Why separate: every config ever published stays readable without the runtime carrying a decade of compatibility branches. But, the legacy code amounted to quite a lot of the final crate size. Disabling it allows on device builds to be smaller.

    The Trainer trait, every concrete *Trainer, TrainerWrapper and the Trainable extension.

    The tokenizers crate re-exports it behind the train feature, which is off by default. It trains only the models whose feature is on (bpe, unigram, wordpiece, wordlevel). Add esaxx_fast for the C++ suffix array behind Unigram training.

    Why separate: training is a batch job on a workstation; inference is a hot loop in a server. They have opposite constraints, so they get opposite dependency budgets.

    cargo run -p bitmap_gen regenerates bitcannon's committed classify tables from unicode-properties, emitting one Atom tag per codepoint.

    Why separate — this is what buys the zero Unicode dependency. unicode-properties is a dependency of this crate and of nothing else: the tables it produces are checked into bitcannon/src/classify/atom_tables.rs as ordinary source, so the Unicode data is resolved once, at development time, by a crate that is never linked into anything that ships and is never published. No build script either — bitcannon compiles with no code generation step, and a release binary contains the tags without containing a Unicode crate to derive them. The release workflow re-runs the generator and fails if the committed table differs, so "baked" cannot silently mean "stale".

    A thin re-export so existing tokenizers::… paths keep working. Depend on this one unless you know you want less.

    Hardware

    No SIMD is required. Every kernel has a portable path that is always compiled and is the byte-exact test oracle for its vectorised siblings, so correctness never depends on a kernel being present — only throughput does.

    opaarch64x86_64wasm32portable fallback
    Unicode atom classifyNEON (baseline)AVX-512 VBMI → SSE4.1/SSSE3, runtime-detectedSIMD128classify_scalar
    bitstream block buildNEONSSE/AVX—build_block_scalar
    literal & added-token scanNEONx86_64—scalar
    vocab bucket nibble matchNEON——scalar

    aarch64 selects at compile time (NEON is baseline). x86_64 dispatches at runtime via is_x86_feature_detected!, so one binary covers every x86_64 CPU since 2008. wasm32 needs the simd128 target feature; without it, the scalar walk.

    Acknowledgements

    This work stands on the shoulders of a lot of open source projects. Thanks to all of them for their great work.

    gigatoken, tiktoken, kitoken, tokie, fastokens, wordchipper and ai-tokenizer each pushed on what a fast tokenizer can be; we read that work, and several of the ideas above reached us because another project showed they were worth trying.

    Docs: guide · quicktour · docs.rs

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    transformers
    huggingface·Python

    统一加载和运行文本、视觉、音频、多模态的预训练模型，覆盖推理与训练，省去为每个模型重写代码的麻烦。

    预训练模型+4训练框架
    168721+10935kv5.22.22026-11-10 03:52 UTC
    LlamaFactory
    hiyouga·Python

    一套统一接口搞定 100+ 大语言模型和多模态模型的微调，支持 LoRA 等高效训练方式，省去为每个模型单独写训练脚本的麻烦。

    微调框架+4训练框架
    75967+449.4kv0.9.52026-11-09 14:55 UTC
    keras
    keras-team·Python

    面向人类友好的深度学习 API，用 Python 编写，可跑在 TensorFlow、JAX、PyTorch 上，帮开发者快速搭建和训练神经网络。

    深度学习+4训练框架
    64419+320kv3.15.12026-11-10 02:55 UTC
    langextract
    google·Python

    用 LLM 从非结构化文本中抽取结构化信息，并给出精确的原文出处定位和交互式可视化，方便核对抽取结果。

    信息抽取+4训练框架
    39297+232.7kv1.7.22026-11-10 01:39 UTC