Tokens can be thought of as pieces of words. Before processing, the input is broken down into tokens that don't necessarily align with word boundaries.
For a detailed explanation of tokens and how to count them, see the OpenAI Tokenizer Guide.
This calculator uses the actual GPT-2 tokenizer (the same one used by many OpenAI models) to provide precise token counts. It:
Note: While this calculator uses the GPT-2 tokenizer, some newer models might use slightly different tokenizers. For model-specific token counts, please use OpenAI's Tokenizer tool.
Note: All calculations are performed locally - no text leaves your browser.
Text processing:
Tool made by Denis Shiryaev, shir-man.com
🏠 Browser-based LLM Playground
Tokens are pieces of words that models process individually. Depending on the tokenizer, tokens may represent whole words, parts of words, or even punctuation.
This calculator uses the GPT-2 tokenizer to provide accurate counts of tokens, words, and characters in real time. All processing is done locally in your browser.
Absolutely. No data leaves your browser; all token calculations are performed locally, ensuring your text remains private.
While this tool uses the GPT-2 tokenizer, which is common among many models, some newer models might use other tokenization methods. For those, the token counts might differ slightly.