Chonkie

lastmod 2024-11-18created 2024-11-181 edits

🦛 CHONK your texts with Chonkie ✨ - The no-nonsense RAG chunking library

A fast, lightweight Python text chunking library for RAG

Features

• Default install size of 21MB, 80-171MB smaller than alternative libraries
• Token chunking 33x faster than other popular libraries
• Supports various chunking strategies such as token, word, sentence, semantic, and SDPM
• Compatible with all major tokenizers such as transformers, tokenizers, and tiktoken
• No external dependencies for the basic features alone

Technical optimizations

• Uses tiktoken, which supports multithreading, for faster tokenization
• Implements aggressive caching and precomputation
• Uses Running Mean Pooling for efficient semantic chunking
• Modular dependency system that lets you install only what you need

See also

• RAG

Favorite site

• Github - bhavnicksm/chonkie
• chonkie · PyPI