When an AI reads or writes, it does not take in whole sentences at once. It works in little pieces called tokens.
A token is a small chunk of text. Words, parts of words, punctuation, and symbols can all be tokens.
For example, "I love robots!" gets broken into pieces like "I", "love", "robot", "s", and "!". The AI reads them one at a time.
Some tokens are whole words, like "dog" or "cool." Longer or trickier words get split into parts, like "play" plus "ground."
Tokens matter because an AI counts them. More tokens means more reading, more cost, and longer answers.
So clear, simple prompts help the robot. Big ideas can be built from little pieces, one token at a time.
What to remember
- A token is a small piece of text.
- An AI reads and writes one token at a time.
- Tokens can be whole words or parts of words.
- More tokens means more reading, cost, and time.
Words to know
- Token
- A small chunk of text an AI processes.
- Tokenization
- Breaking text into tokens.
- Prompt
- The text you give the AI to read.
- Context
- All the tokens the AI can keep in mind at once.
For grown-ups
Models operate on tokens, sub-word units produced by a tokenizer, not raw characters or whole words. Token counts drive context limits, latency, and cost. Understanding tokenization explains quirks like odd word splits, and why concise prompts and outputs are cheaper and faster.
Want the full story? These go deeper:
