Tokens that mean something special
Beyond ordinary text pieces, tokenizers reserve special tokens that carry structural meaning rather than literal characters. Common examples include beginning of sequence, end of sequence, padding, and unknown.
Roles and turns
Chat models add role markers that delimit who is speaking. A chat template is a formatting rule that wraps each message with these markers so the raw text stream encodes a structured dialogue.
- A system message sets behavior.
- User and assistant turns alternate.
- Turn boundaries are marked by special tokens the model learned during fine tuning.
Why templates must match training
The model only behaves well if you format prompts with the exact template used during its fine tuning. Wrong markers, missing role tokens, or a stray space can degrade responses or cause the model to keep talking past its turn.
A safety surface
Special tokens are also a boundary attackers probe. If user text can inject control tokens, it may spoof a system or assistant turn, so good tokenizers prevent ordinary text from producing reserved ids.
Key idea
Special tokens encode structure like turn boundaries, and you must apply a model's exact chat template for it to behave as trained.