Meaning becomes a direction
An embedding model turns text into a list of numbers. Texts about the same thing point the same way, even when they share no words.
How does a model answer from documents it was never trained on?
Nobody teaches it the documents. Before it answers, a search finds the few passages that matter and pastes them into the prompt. The model reads them like any other text, and cites them.
Embeddings here have nine named dimensions instead of hundreds of learned ones, and each chunk is one sentence.
First the shop's documents are split into chunks and each becomes a vector. A question becomes a vector too, the closest chunks are pasted into the prompt, and the model answers from them, with a source.
Ready7 chunks indexed. Ask a question.
Ready. 7 chunks indexed. Ask a question.
Ask
Retrieval
Documents
Nearest chunksbold: used · faint: under 0.3
| chunk | text | score |
|---|
Counters
7Chunks indexed
0Questions
0Answers with a source
0“Not in the docs”
0Answers with no source
0Out-of-date answers
Sheet 09 · From question to cited answerThree documents, one chunk per sentenceNine-dimension embeddings
Three ideas explain most of it
Search first, then ask
The question is embedded the same way and the nearest chunks are fetched. Only those few go to the model, so the documents can be far bigger than its context window.
The prompt does the teaching
The model gets an instruction to answer only from the passages it was given, and to say so when they don't cover the question. Nothing about the model itself changes.
RAG is a prompt, assembled by a search
Live: the prompt for the last question
$ cat prompt.txt
(no question yet)Everything the model knows about this shop arrives in that prompt. Ask a question and watch it fill: the chunks above, the question below, and an instruction that makes “I don't know” an acceptable answer.
That's why RAG beats retraining for this job. Change a document and re-index it, and the next answer changes, with a citation someone can check.
It's also why retrieval quality is the whole game. The model can only be as right as the passages it was handed. A wrong chunk gives a confident wrong answer; a missing chunk gives a guess, or, if the prompt allows it, an honest refusal.
Most failures are search failures
The copy falls behind the source
The index is a snapshot. If documents change and it isn't rebuilt, answers stay confidently cited and quietly wrong.
The right passage, never found
Keyword search misses paraphrases. Vector search can miss exact names, codes and numbers. Many systems combine both and re-rank the results.
Chunks cut in the wrong place
Split a table from its heading, or a rule from its exception, and each chunk is true on its own but misleading alone. How you chunk decides what can be found.
Two predictions
The returns policy changes from 30 days to 14 in the document. Nobody re-indexes. A customer asks how long they have. What do they hear?
Someone asks “When will I get my money back?”. The document says “Refunds reach your card in 5 to 7 working days.” Which search finds it?
What this model leaves out
Real embeddings
Real embedding models produce hundreds or thousands of dimensions, none of which has a name. They're learned from huge amounts of text, which is how “money back” ends up near “refund” without anyone listing synonyms.
Approximate search
Comparing a question with every chunk is fine for eight chunks, not eight million. Vector databases use approximate indexes such as HNSW that check a small fraction of them and are almost always right.
Chunk size and overlap
Chunks are usually a few hundred tokens, often overlapping so a sentence near a boundary appears in both. Here each sentence is one chunk, to keep scores readable.
Writing the answer
Here the model quotes its best chunk. A real one writes a fresh answer from all the chunks it was given, and can still misread them, so citations matter.