A pure Javascript tokenizer running in your browser that can load tokenizer.json and
tokenizer_config.json from any repository on Huggingface. You can use it to count tokens and
compare how different large language model vocabularies work. It's also useful for debugging prompt templates.
Implementation details
Parallel model loading: All tokenizers load simultaneously using Promise.all() instead of sequentially, to improve startup time
Progressive rendering: Models appear and update individually as they finish loading, providing immediate feedback
Debounced input processing: Text changes are debounced by 300ms to prevent excessive re-tokenization during typing
Client-side tokenization: Uses transformers.js for pure browser-based tokenization without server dependencies
Ruby annotations: Tokens are displayed using HTML <ruby> elements with text above and token numbers below
Space preservation: Automatically detects and removes tokenizer space-stripping to accurately show whitespace tokens
URL parameter support: Share configurations via ?text=...&models=... for easy collaboration
LocalStorage persistence: Model lists persist across browser sessions with automatic fallback to defaults
Memory management: Models are cached in memory and only loaded once, with cleanup on deletion
Dark mode support: Automatic theme detection with appropriate colour schemes for all UI elements
Usage
Adding models: Copy model names from HuggingFace (e.g., from the title of model pages like "microsoft/Phi-3-mini-4k-instruct") and paste into the input field
Deleting models: Click the red "🗑️ Delete" button next to any model. You'll get a confirmation prompt and cannot delete the last model
Sharing configurations: Click the "📋 Share" button to copy a URL containing your current text and model selection
URL parameters: Share specific setups using ?text=your%20text&models=model1,model2,model3 format
Text input: Type or paste text into the textarea - tokenization happens automatically with a short delay
Reading tokens: Each word/subword piece shows the original text above and the token number below
Token colours: Different background colours help distinguish adjacent tokens, cycling through 10 colours
Model comparison: Compare how different models tokenize the same text - useful for understanding vocabulary differences
Keyboard shortcuts: Press Enter in the "Add model" field to quickly add new tokenizers
Error recovery: Failed models show detailed error messages but don't break the application
Mobile usage: Fully functional on mobile devices with responsive design adaptations
Offline capability: Once models are loaded, tokenization works completely offline
If you are wondering why are there so many models under Xenova, it's because they work for HuggingFace and re-upload just the tokenizers, so it's possible to load them without agreeing to model licences.
