Llama-tokenizer-js
lastmod 2023-06-27created 2023-06-271 edits
JS tokenizer for LLaMA based LLMs
A LLaMA tokenizer that runs in the browser
Features
- Can be used to count tokens accurately on the client side
- Easy to use as a single file with zero dependencies
- Compatible with most LLaMA-based models
- Can tokenize about 2,000 tokens in under 20 ms