Count tokens for Llama 3 and Llama 2 using the models' real tokenizers, running entirely in your browser, with token boundaries you can see.
Your text never leaves your browser. The tokenizer itself is downloaded once from the Hugging Face CDN.
-
-
0
0
Llama 3
The genuine article: Llama 3 models (3, 3.1, 3.2, 3.3) share a 128K-vocabulary BPE tokenizer, and Llama 2 uses a 32K SentencePiece tokenizer. Both run in your browser via transformers.js, so counts match what the models actually see.
Llama 3's vocabulary is four times larger, so it packs the same text into fewer tokens. The same document can shrink by 15 to 25 percent in token count moving from Llama 2 to Llama 3.
No. Tokenization runs entirely in your browser. The only network request is downloading the open tokenizer files once from the Hugging Face CDN.
The GPT column uses OpenAI's open o200k tokenizer for comparison. For exact Claude counts, use our Claude Tokenizer; for Gemini, our Gemini Tokenizer.
MemoryPlugin gives ChatGPT, Claude, Gemini, and 20+ other AI tools one shared long-term memory and a searchable chat history. Tell one AI something once, and the rest have it from your next conversation.
Try MemoryPlugin free