Text goes in as numbered pieces
A tokenizer splits text into common words and fragments of rare ones. Context limits, prices and speeds are all measured in tokens, and a word can cost several.
How does a language model write a sentence it has never seen?
It doesn't write sentences. It reads everything so far, scores every possible next token, picks one, and starts again. Everything else, from memory to mistakes, follows from that loop.
A toy model: a small table of continuations stands in for billions of weights, and only the top few candidates exist.
The model never writes a sentence. It scores every possible next token, one is picked, it's appended, and the whole context goes through again. “Paris.” takes three passes.
ReadyPick a prompt and watch it become one token at a time.
Ready. Pick a prompt and watch it become one token at a time.
Prompt
Generate
Sampling
The last forward pass␣ marks a leading space
| next token | model | sampler draws from |
|---|
Counters
0Prompt tokens
0Forward passes
0In the window now
0Fell out of the window
0Not the top choice
0Picks under 50%
Sheet 07 · From prompt to next tokenA toy model with a tiny vocabularyProbabilities illustrative
Three ideas explain most of it
Every token is a full pass
The model reads the whole context and returns a probability for every token in its vocabulary. One is picked and appended, and the model runs again from the start.
Randomness is added afterwards
The model is a fixed function: the same context always gives the same scores. Variety comes from the sampler, which temperature makes bolder or more cautious.
The context window is the only memory
Live: the window, token by token
$ context --window 64 (0 in, 0 dropped) (empty)
A chat feels like a conversation, but the model keeps nothing between calls. The app sends the whole conversation again every time, and the model reads it from the first token.
The window has a fixed size. When a conversation outgrows it, something has to go, usually the oldest messages. Apps hide this by summarising, or by retrieving the relevant parts again: that's what RAG is for.
“Remembering” within the window is attention: the model can look back at any token it can see. In the recall scenario it finds “ name is” earlier on and copies what came after it.
Fluent is not the same as correct
Missing context, confident answer
The model can't tell that something fell out of its window. It sees a question about a name and gives the most likely one.
Low confidence, written plainly
When nothing it learned fits, the probabilities are flat, but a token is still picked. The text reads as sure as any other. This is what people call hallucination.
One unlucky draw steers the rest
Each token is conditioned on the ones before it. An unlikely early pick isn't corrected later: the model carries on, consistently, from where it landed.
Two predictions
The conversation starts with “My name is Barry Allen” and ends with “What is my name?”. The window holds 16 tokens and the conversation is 29. What does the model answer?
Temperature 0. You send “The capital of France is” twice, with a different random seed each time. Do the answers differ?
What this model leaves out
Real weights and vocabularies
A real model computes its scores with billions of learned numbers, over a vocabulary of 50,000 to 200,000 tokens, every one of which gets a probability on every pass. Here a small table of continuations plays that part.
Attention
Copying the token that followed an earlier match is one thing attention learns to do. Real attention weighs every token against every other in many layers at once, and does far more than copy.
Top-p, top-k and caching
Real samplers also cut off the long tail of unlikely tokens before drawing. And servers cache the work done on earlier tokens, so each new pass is cheaper than reading the whole context from scratch.
Training and chat tuning
This page shows inference only. Saying “I don't know”, following instructions and refusing come from training after pre-training. They change the probabilities, not the loop.
Connected mechanisms
Artificial IntelligenceRAGHow can an AI answer questions using external knowledge?Interactive · 6 min →Artificial IntelligenceAI agentsWhat happens when an AI can choose tools and take actions?Interactive · 7 min →Artificial IntelligenceAI automationsHow does a workflow like n8n turn an email into an action?Interactive · 6 min →