TokenDagger

lastmod 2025-07-02created 2025-07-021 edits

A tokenizer faster than OpenAI's Tiktoken

About

• A high-performance tokenizer 100% compatible with OpenAI's TikToken, delivering more than 2x throughput on large-scale text processing and 4x faster code tokenization
• Maximizes token pattern matching speed with a fast PCRE2-based regular expression parsing engine
• A simplified BPE algorithm minimizes performance degradation when handling large numbers of special tokens
• In real benchmarks, code tokenization is more than 4x faster, and it can be used as a drop-in replacement in existing TikToken code
• Supports Python 3.8+, easy to install from PyPI with pip install tokendagger, and depends on PCRE2

See also

• OpenAI

Favorite site

• Github - M4THYOU/TokenDagger
• TokenDagger – A tokenizer faster than OpenAI's Tiktoken | GeekNews