Early build · scores computed by the deterministic engine from real, sourced developments (30 days) · weights v1.0-prior
LatestLearn › Basics
Vikshy Learn · Basics

Token

The small chunk of text an AI reads and writes, often part of a word.

AI models do not see whole words the way we do. They break text into little pieces called tokens, which can be a short word, part of a longer word, or even a single punctuation mark. Everything the model reads and everything it writes is counted in tokens, so tokens are the basic unit of an AI's attention and, usually, of its cost.

For example, The word 'unbelievable' might be split into three tokens: 'un', 'believ', and 'able'.

← All 42 concepts in the glossary