A tokenizer running entirely in your browser. Count tokens and compare how different large language model vocabularies split the same text.

Above: the same five pieces in three vocabularies, with a different ID every time.
Below: loads tokenizer.json and tokenizer_config.json from any repository on Hugging Face. Also handy for debugging prompt templates.

Re-tokenises as you type

Token count

All models
Usage
  • Adding models: Copy model names from Hugging Face (e.g., from the title of model pages like "microsoft/Phi-3-mini-4k-instruct") and paste into the input field
  • Deleting models: Click the bin button on any model card. You'll get a confirmation prompt and cannot delete the last model
  • Sharing configurations: Click "Copy share link" to copy a URL containing your current text and model selection. The text is compressed into the part after the #, so even long texts make a working link
  • URL parameters: Write links by hand using the ?text=your%20text&models=model1,model2,model3 format
  • Text input: Type or paste text into the text box - tokenization happens automatically with a short delay. The text is kept when you reload, until you close the tab
  • Reading tokens: Each word/subword piece shows the original text above and the token number below
  • Token colours: Different background colours help distinguish adjacent tokens, cycling through 10 colours
  • Newlines: Tokens containing a line break are shown with a ↵ symbol
  • Model comparison: Compare how different models tokenize the same text - click a name in the token count list to jump to its card, and "All models" to come back
  • Keyboard shortcuts: Press Enter in the "Add from Hugging Face" field to quickly add new tokenizers
  • Clean-up: Tokens show their exact text, so " ." keeps its space. Tick "Clean up spaces before punctuation" on a card to see it the way the model's decoder would tidy it (".", "don't"), next to that model's own default. Only the text changes, never the IDs or the count
  • Error recovery: Failed models show detailed error messages but don't break the application
  • Mobile usage: Fully functional on mobile devices with responsive design adaptations
  • Offline capability: Once models are loaded, tokenization works completely offline
  • If you are wondering why are there so many models under Xenova, it's because they work for Hugging Face and re-upload just the tokenizers, so it's possible to load them without agreeing to model licences.
Implementation details
  • Rust tokenizers in WebAssembly: Tokenization uses Hugging Face's tokenizers library compiled to WebAssembly, the same code that runs in Python and Rust
  • Web Workers: Each tokenizer loads and runs in its own worker, in parallel and off the main thread, so typing stays smooth. The WebAssembly module is compiled once and shared
  • Progressive rendering: Models appear and update individually as they finish loading, providing immediate feedback
  • Debounced input processing: Text changes are debounced by 300ms to prevent excessive re-tokenization during typing, and results for text that has since changed are dropped
  • Ruby annotations: Tokens are displayed using HTML <ruby> elements with text above and token numbers below
  • Space preservation: Automatically detects and removes tokenizer space-stripping to accurately show whitespace tokens
  • Share links: The text is compressed with CompressionStream('deflate-raw') into #text=..., which browsers never send to the server, so links don't run into its URL length limit. ?text=...&models=... also works
  • Storage: Model lists persist across browser sessions in localStorage with automatic fallback to defaults. The text is kept in sessionStorage, so it survives a reload but not closing the tab
  • Caching: Tokenizer files are kept with the Cache API, so reloading the page doesn't download them again. Deleting a model stops its worker
  • Dark mode support: Automatic theme detection with appropriate colour schemes for all UI elements