After this lesson you can explain how a system finds passages before it answers, and why it can still misquote them.
About 10 minutes, or 16 with the Deep layers
Reading depth for every section
In short
A RAG system first finds the passages most like your question. Then a language model writes an answer from them.
A close match is not a right answer. The answer can still change a word, or add a claim the passage never made.
Read the passage beside the answer. Compare them sentence by sentence.
Reading depth forListen to
Why give an AI your documents
A language model is a program trained on a large amount of text. It works out chances for the next piece of text. Chatbots run on large ones.
A model learns from the text it was trained on. Your private documents may not be in that text. How do you get answers from them?
In 2020, Patrick Lewis and eleven co-authors wrote about this problem. They said these models keep facts inside the model itself. They said updating those facts, and showing where an answer came from, were still open problems.
Their proposal was to give the model a store of text to look in first. They called this retrieval-augmented generation, or RAG.
A maker can say when a model’s training text ends. Anthropic is an AI company, and Claude Sonnet 5.5 is one of its models. For it, Anthropic lists a “reliable knowledge cutoff” and a “training data cutoff”. Both are June 2026. Lesson 1.5, Where your question goes, covers it.
The paper says large models trained in advance “store factual knowledge in their parameters”. Here the parameters are the numbers a model learns during training. It says “their ability to access and precisely manipulate knowledge is still limited”. It adds that “providing provenance for their decisions and updating their world knowledge remain open research problems.” That is how its authors described the field when they posted the paper on 22 May 2020.
The paper’s RAG models combine two kinds of memory. The first is the model’s own numbers, which the paper calls parametric. The second is a store of text that the system can search, which it calls non-parametric.
In the paper’s tests, the RAG models did better than earlier systems on three question-answering tests. For language generation, they made “more specific, diverse and factual language” than a model with no store. Those are results for that paper’s models, on its tests, in 2020. They do not show that every system called RAG is reliable. The sections below show where one can fail.
Step one: find the passages
A RAG system works in two steps. First it finds passages. Then a language model writes an answer from them. Finding passages is called retrieval.
A passage is a short piece of a longer text. The 2020 paper cut Wikipedia articles into pieces of 100 words. Wikipedia is a free online encyclopedia. The paper called the pieces chunks. In this lesson we say passage.
Retrieval does not look up one stored answer. It finds passages. The model then writes new text from them.
In the paper, the store was one copy of Wikipedia, from December 2018. It held 21 million pieces of 100 words each. For each question, the system retrieved the top few pieces. The authors used five or ten while training the model.
In the paper, the store was only as current as that copy. In our own test, the models read the passages we sent. Nothing in our setup checked whether a passage was true or up to date.
What “similar” means
How does the system pick passages? It uses embeddings. An embedding is a list of numbers that stands for a piece of text. A model makes it.
Pieces that are alike should get lists that sit close together.
The glossary is the list of terms and meanings in our lessons. In our activity, the store of text is a fixed snapshot of it. It holds 159 entries from the first 25 lessons, as they stood on 9 October 2026. Each entry is one passage.
A list is made for the question too. The passages whose lists sit closest are picked.
The similarity is a score for how close two lists are. In this lesson it is the cosine of the angle between the lists. That number is higher when the lists point more nearly the same way.
The number says how alike two lists are. It does not say that a passage is true. It does not say that a passage answers the question.
Google’s course says an embedding is “a vector representation of data in embedding space”, and that embeddings “place similar items closer together”. A vector is a list of numbers. Its page on similarity lists three ways to score closeness: Euclidean distance, cosine and dot product. For cosine, the score “increases” as similarity increases.
We made our lists once, on 9 October 2026, with a model called nomic-embed-text. It has 137 million parameters, an Apache 2.0 licence and 768 numbers in each list. We made one list for every entry in that snapshot, and one for each of eight questions. The page never runs the model.
Our words “very close”, “close” and “not close” are ours. We set the cut-offs by looking at the eight questions. A raw cosine of 0.80 or more is “very close”. From 0.65 up to 0.80 is “close”.
Here is a case where the right passage exists and is not found. Question 2 asks what temperature changes in a language model. In lesson 1.4, The next word, temperature is a setting that changes how even the chances are before a piece is picked. The snapshot has an entry for it. It ranked sixth, with a raw cosine of 0.64. The three entries we sent to the models scored 0.76, 0.75 and 0.71. None of them was about temperature.
Find, then answer
The activity Find, then answer has eight questions. You pick one. You predict whether the first match is the entry for a term the question asks about. Question 4 asks about two, round and epoch. You also predict whether both answers will stay inside the passages.
Staying inside means every claim and quote comes from the passages and nothing is added.
The questions use some terms that may be new:
Temperature: a setting that controls how random a model’s output is.
Tag characters: invisible format characters in Unicode, a standard that gives every character a number. They are used to build tags.
Epoch: one full pass through the examples used to train a model. Our glossary also calls it a round.
Context window: the amount of text a model can look back on when it writes.
Citation: a note in an answer that points to its source.
AI Act: a law of the European Union (EU) about AI. A deployer is whoever uses an AI system under its authority. Personal, non-professional use is left out.
Fine-tuning: further training of a language model on more data.
Then you see the five closest entries. Each one shows how close it is, in words, and the lesson it comes from. After that you see the answers two models wrote from the top three, with anything that goes beyond them marked.
Very close, close and not close are our own labels for ranges of the score. One question asks about fine-tuning, which the snapshot lacks. Five entries still come back, and none is close. Lessons 5.1 and 5.6 do have entries for it, but they are not in the snapshot.
Nothing is counted or saved. The answers were recorded on 8 October 2026.
The snapshot is our own text, from the lessons on this site. Some terms appear more than once, in different lessons, with slightly different wording. For example, the question “What is a token?” finds five entries for token, from five lessons, worded a little differently. In this store, near-copies like these can fill several of the top matches.
The eighth question, “What is fine-tuning?”, is about a term the snapshot does not cover. The system still returns five entries. All five are “not close”. A system that always returns its top few passages will return some, even when none is about the question.
Try it: Find, then answer
This system first finds the passages most like your question. The store is a snapshot of this site's glossary. Then a model answers from them.
What the recorded answers showed
We sent eight questions to two models. Each question went with the three passages our own code found for it. That makes sixteen answers, recorded on 8 October 2026.
We checked every quote and every sentence against the three passages.
Claude Sonnet 5.5: none of the eight answers changed a quote or went beyond its passages.
Ministral 3 14B: four of the eight answers did.
Question 1: one quote is altered, by one comma.
Question 3: one quote is altered. The answer also says all tag characters are used to end a tag. The passage says only the last one ends a tag.
Both question 3 answers say Unicode has 96 tag characters. The passage says 96, and the answers match it. Unicode counts 97. Staying inside the passage is not the same as being true.
Question 4: two quotes are altered. They begin with the passage number and the term, such as "[1] epoch:". They leave out the lesson name that the passage holds. The words of the entry itself match.
Question 7: the answer adds that deployers are typically professional or organizational users. The passage does not say that.
Questions 2, 5 and 8 had passages that did not hold the answer. All six answers said so.
For question 2, the snapshot does have an entry on what the question asks about. Our retrieval ranked it sixth, so it was not among the three we sent. That is a miss in finding, not in the answer.
We told both models to use only the passages. Twelve of the sixteen answers did. Four did not.
This is eight questions, one answer each, from two models on one day. It does not show how often any system misquotes.
An answer is marked when a word or punctuation mark inside quote marks is added, left out or changed. It is also marked when it says something the passages do not say, even if that is true.
We ignore the quote marks, spacing and a capital letter at the start of a quote. We also ignore a comma or full stop just inside the closing quote mark.
By this rule four quotes were altered and two statements went beyond the passages, in four answers in all. A citation number placed after a sentence it does not fully cover is noted, not counted.
The passages are our own short glossary entries, and they leave things out. An answer built from a store can repeat the store's gaps.
Question 4: the round entry says training changes the model's numbers after each round. Google's glossary says one full pass is made of training iterations, and each iteration updates the numbers. So the numbers can change many times in one round.
Question 7: the provider entry leaves out a body that puts a system into service under its own name. The high-risk entry leaves out the exceptions in Article 6(3) of the AI Act.
What is a token?
The five closest entries
token, from the lesson "Attention and the context window": very close
token, from the lesson "Context": very close
token, from the lesson "Examples": very close
token, from the lesson "The next word": very close
token, from the lesson "Tokens": very close
Raw cosine
The raw number behind each word, to two places. It compares two lists of numbers. It does not say the passage answers the question.
token: 0.88
token: 0.88
token: 0.88
token: 0.86
token: 0.85
First match. Our reading: Yes. The first match is the entry for token.
The answer built from the top three entries
Claude Sonnet 5.5 and Ministral 3 14B, recorded 8 October 2026. Told: use only the passages, number each sentence, quote at least once, say so if the passages do not answer.
These runs are written out in full at the end of this activity.
Stays inside the passages.
Our reading, Claude Sonnet 5.5: Yes.
Our reading, Ministral 3 14B: No.
What does temperature change in a language model?
The five closest entries
language model, from the lesson "Tokens": close
language model, from the lesson "The next word": close
attention, from the lesson "Attention and the context window": close
base model, from the lesson "Why they make things up": close
token, from the lesson "Tokens": not close
Raw cosine
The raw number behind each word, to two places. It compares two lists of numbers. It does not say the passage answers the question.
language model: 0.76
language model: 0.75
attention: 0.71
base model: 0.66
token: 0.64
First match. Our reading: No. The first match is language model. The entry for temperature is number 6 in the full ranking: not close.
The answer built from the top three entries
Claude Sonnet 5.5 and Ministral 3 14B, recorded 8 October 2026. Told: use only the passages, number each sentence, quote at least once, say so if the passages do not answer.
These runs are written out in full at the end of this activity.
Stays inside the passages.
Our reading, Claude Sonnet 5.5: Yes.
Our reading, Ministral 3 14B: Yes.
How many tag characters are there in Unicode, and what are they used for?
The five closest entries
tag character, from the lesson "Context": very close
Unicode, from the lesson "Context": close
code point, from the lesson "Context": close
XML tag, from the lesson "Context": close
XML tag, from the lesson "A skill is a written procedure (SOP)": close
Raw cosine
The raw number behind each word, to two places. It compares two lists of numbers. It does not say the passage answers the question.
tag character: 0.87
Unicode: 0.73
code point: 0.69
XML tag: 0.69
XML tag: 0.68
First match. Our reading: Yes. The first match is the entry for tag character.
The answer built from the top three entries
Claude Sonnet 5.5 and Ministral 3 14B, recorded 8 October 2026. Told: use only the passages, number each sentence, quote at least once, say so if the passages do not answer.
These runs are written out in full at the end of this activity.
Stays inside the passages.
Our reading, Claude Sonnet 5.5: Yes.
Our reading, Ministral 3 14B: No.
What is the difference between a round and an epoch in training?
The five closest entries
epoch, from the lesson "Training": close
round, from the lesson "Training": close
training, from the lesson "Training": close
training, from the lesson "Environmental cost": not close
training cut-off, from the lesson "Where your question goes": not close
Raw cosine
The raw number behind each word, to two places. It compares two lists of numbers. It does not say the passage answers the question.
epoch: 0.75
round: 0.74
training: 0.68
training: 0.62
training cut-off: 0.62
First match. Our reading: Yes. The first match is the entry for epoch.
The answer built from the top three entries
Claude Sonnet 5.5 and Ministral 3 14B, recorded 8 October 2026. Told: use only the passages, number each sentence, quote at least once, say so if the passages do not answer.
These runs are written out in full at the end of this activity.
Stays inside the passages.
Our reading, Claude Sonnet 5.5: Yes.
Our reading, Ministral 3 14B: No.
How many tokens fit in the context window of Claude Sonnet 5?
The five closest entries
context window, from the lesson "Attention and the context window": close
model, from the lesson "Bad and good prompts": close
token, from the lesson "Examples": not close
token, from the lesson "Tokens": not close
token, from the lesson "The next word": not close
Raw cosine
The raw number behind each word, to two places. It compares two lists of numbers. It does not say the passage answers the question.
context window: 0.70
model: 0.65
token: 0.61
token: 0.60
token: 0.60
First match. Our reading: Yes. The first match is the entry for context window.
The answer built from the top three entries
Claude Sonnet 5.5 and Ministral 3 14B, recorded 8 October 2026. Told: use only the passages, number each sentence, quote at least once, say so if the passages do not answer.
These runs are written out in full at the end of this activity.
Stays inside the passages.
Our reading, Claude Sonnet 5.5: Yes.
Our reading, Ministral 3 14B: Yes.
How much of a page's text does a citation show?
The five closest entries
citation, from the lesson "Where your question goes": close
citation, from the lesson "Why they make things up": close
context window, from the lesson "Attention and the context window": close
attention, from the lesson "Attention and the context window": not close
disclosure line, from the lesson "Work, creativity and credit": not close
Raw cosine
The raw number behind each word, to two places. It compares two lists of numbers. It does not say the passage answers the question.
citation: 0.77
citation: 0.67
context window: 0.65
attention: 0.63
disclosure line: 0.63
First match. Our reading: Yes. The first match is the entry for citation.
The answer built from the top three entries
Claude Sonnet 5.5 and Ministral 3 14B, recorded 8 October 2026. Told: use only the passages, number each sentence, quote at least once, say so if the passages do not answer.
These runs are written out in full at the end of this activity.
Stays inside the passages.
Our reading, Claude Sonnet 5.5: Yes.
Our reading, Ministral 3 14B: Yes.
Under the AI Act, who counts as a deployer?
The five closest entries
deployer, from the lesson "The law: AI rules in Ireland and the EU": very close
provider, from the lesson "The law: AI rules in Ireland and the EU": close
high-risk AI system, from the lesson "The law: AI rules in Ireland and the EU": close
AI literacy, from the lesson "The law: AI rules in Ireland and the EU": not close
human oversight, from the lesson "What trustworthy AI needs": not close
Raw cosine
The raw number behind each word, to two places. It compares two lists of numbers. It does not say the passage answers the question.
deployer: 0.88
provider: 0.75
high-risk AI system: 0.68
AI literacy: 0.65
human oversight: 0.64
First match. Our reading: Yes. The first match is the entry for deployer.
The answer built from the top three entries
Claude Sonnet 5.5 and Ministral 3 14B, recorded 8 October 2026. Told: use only the passages, number each sentence, quote at least once, say so if the passages do not answer.
These runs are written out in full at the end of this activity.
Stays inside the passages.
Our reading, Claude Sonnet 5.5: Yes.
Our reading, Ministral 3 14B: No.
What is fine-tuning?
The five closest entries
attention, from the lesson "Attention and the context window": not close
regulation, from the lesson "The law: AI rules in Ireland and the EU": not close
made-up detail, from the lesson "Why they make things up": not close
oversight, from the lesson "Harm and misuse": not close
human oversight, from the lesson "What trustworthy AI needs": not close
Raw cosine
The raw number behind each word, to two places. It compares two lists of numbers. It does not say the passage answers the question.
attention: 0.54
regulation: 0.53
made-up detail: 0.53
oversight: 0.53
human oversight: 0.53
First match. Our reading: No. The snapshot of the glossary has no entry for fine-tuning. The first match is attention: not close.
The answer built from the top three entries
Claude Sonnet 5.5 and Ministral 3 14B, recorded 8 October 2026. Told: use only the passages, number each sentence, quote at least once, say so if the passages do not answer.
These runs are written out in full at the end of this activity.
Stays inside the passages.
Our reading, Claude Sonnet 5.5: Yes.
Our reading, Ministral 3 14B: Yes.
The two AIs and their settings
Claude Sonnet 5.5
claude-sonnet-5-5, on Anthropic Messages API. Recorded on 8 October 2026.
Longest answer, in tokens: 4000
Temperature: not sent, and the value the service used was not recorded
Tools: no tools
Instructions from us behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
Ministral 3 14B
ministral-14b-2512, on Mistral AI API. Recorded on 8 October 2026.
Model name sent: ministral-14b-2512
Longest answer, in tokens: 4000
The model's default temperature, read from its model list: 0.3
Temperature: not sent, and the value the service used was not recorded
Tools: no tools
Instructions from us behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
Show all runs
Question 1: What is a token?
Claude Sonnet 5.5 Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
stayed inside the passages. Both quotes match, and it says the second passage repeats the first.
You sent:Passages:
[1] token (from the lesson "Attention and the context window"): A small piece of text, often part of a word, that a language model works with (lesson 1.3).
[2] token (from the lesson "Context"): A small piece of text, often part of a word, that a language model works with (lesson 1.3).
[3] token (from the lesson "Examples"): A small piece of text that an AI reads, such as a word, part of a word or a letter.
Question: What is a token?
A token is "a small piece of text, often part of a word, that a language model works with" [1]. The same definition appears in another lesson [2]. A third passage describes it slightly differently, as "a small piece of text that an AI reads, such as a word, part of a word or a letter" [3].
Ministral 3 14B Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
one quote is altered. A comma was added before "or a letter". Passage 3 reads "such as a word, part of a word or a letter", with no comma before "or". This changes the quote. It does not make the definition false.
You sent:Passages:
[1] token (from the lesson "Attention and the context window"): A small piece of text, often part of a word, that a language model works with (lesson 1.3).
[2] token (from the lesson "Context"): A small piece of text, often part of a word, that a language model works with (lesson 1.3).
[3] token (from the lesson "Examples"): A small piece of text that an AI reads, such as a word, part of a word or a letter.
Question: What is a token?
The passages describe a token as "a small piece of text, often part of a word, that a language model works with" [1].
They also define it as "a small piece of text, often part of a word, that a language model works with" [2].
Additionally, it is described as "a small piece of text that an AI reads, such as a word, part of a word, or a letter" [3].
Question 2: What does temperature change in a language model?
Claude Sonnet 5.5 Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
said the passages do not answer. True for these passages.
You sent:Passages:
[1] language model (from the lesson "Tokens"): A program trained on a large amount of text that works out a likely next piece of text. Chatbots like ChatGPT run on large ones.
[2] language model (from the lesson "The next word"): A program trained on a large amount of text that works out chances for the next piece of text. Chatbots like ChatGPT run on large ones.
[3] attention (from the lesson "Attention and the context window"): In a language model, the weighing of earlier pieces of text, done with numbers, to write the next piece. The word is borrowed from people.
Question: What does temperature change in a language model?
The passages do not say what temperature changes in a language model. They define a language model [1][2] and attention [3], but none of them mentions temperature, so I can't answer the question from them.
Ministral 3 14B Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
said the passages do not answer. True for these passages.
You sent:Passages:
[1] language model (from the lesson "Tokens"): A program trained on a large amount of text that works out a likely next piece of text. Chatbots like ChatGPT run on large ones.
[2] language model (from the lesson "The next word"): A program trained on a large amount of text that works out chances for the next piece of text. Chatbots like ChatGPT run on large ones.
[3] attention (from the lesson "Attention and the context window"): In a language model, the weighing of earlier pieces of text, done with numbers, to write the next piece. The word is borrowed from people.
Question: What does temperature change in a language model?
The passages provided do not answer the question about what temperature changes in a language model.
Question 3: How many tag characters are there in Unicode, and what are they used for?
Claude Sonnet 5.5 Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
gave the count and the range, and said the passages say no more about use. Every quote matches. It repeated the passage's count of 96, which is not Unicode's count.
You sent:Passages:
[1] tag character (from the lesson "Context"): One of 96 Unicode characters, from U+E0020 to U+E007F. The first 95 match plain ASCII characters. The last one ends a tag.
[2] Unicode (from the lesson "Context"): A standard that gives every character a number, so different computers can share text.
[3] code point (from the lesson "Context"): A character's number in Unicode, written like U+200B.
Question: How many tag characters are there in Unicode, and what are they used for?
Unicode has 96 tag characters, ranging from U+E0020 to U+E007F [1]. According to the passage, "The first 95 match plain ASCII characters" [1]. The last one "ends a tag" [1]. The passages say nothing more about what tag characters are used for, so I can't give a fuller answer [1].
Not true: The Unicode Standard says its Tags block "encodes a set of 97 special-use tag characters". One of them, U+E0001, is deprecated. The 96 in the passage run from U+E0020 to U+E007F. The part marked: "Unicode has 96 tag characters"
Ministral 3 14B Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
one quote is altered and one claim goes beyond the passage. The passage says "ends a tag", and says only that the last character ends a tag. The model wrote "end a tag" and said all of them are used to end a tag.
repeated the passage's count of 96, which is not Unicode's count.
You sent:Passages:
[1] tag character (from the lesson "Context"): One of 96 Unicode characters, from U+E0020 to U+E007F. The first 95 match plain ASCII characters. The last one ends a tag.
[2] Unicode (from the lesson "Context"): A standard that gives every character a number, so different computers can share text.
[3] code point (from the lesson "Context"): A character's number in Unicode, written like U+200B.
Question: How many tag characters are there in Unicode, and what are they used for?
There are 96 tag characters in Unicode, specifically from U+E0020 to U+E007F ["One of 96 Unicode characters, from U+E0020 to U+E007F" [1]].
They are used to "end a tag" ["The last one ends a tag" [1]]. The first 95 of these characters "match plain ASCII characters" ["The first 95 match plain ASCII characters" [1]].
Not true: The passage says "The last one ends a tag", one character and not all of them. The Unicode Standard says tag characters are encoded "to enable the construction of tags". The part marked: "They are used to "end a tag""
Not true: The Unicode Standard says its Tags block "encodes a set of 97 special-use tag characters". One of them, U+E0001, is deprecated. The 96 in the passage run from U+E0020 to U+E007F. The part marked: "There are 96 tag characters in Unicode"
Question 4: What is the difference between a round and an epoch in training?
Claude Sonnet 5.5 Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
said the passages give no difference. Every quote matches.
You sent:Passages:
[1] epoch (from the lesson "Training"): Google's word for one round, one pass over all the examples.
[2] round (from the lesson "Training"): One pass over all the examples. After each round, training changes the model's numbers a little.
[3] training (from the lesson "Training"): Changing a model's numbers, round after round, so its guesses on the examples get less wrong.
Question: What is the difference between a round and an epoch in training?
The passages give no difference in meaning: an epoch is "Google's word for one round, one pass over all the examples." [1] A round is defined in the same way, as "One pass over all the examples." [2] So "epoch" is simply Google's name for a round. [1]
Ministral 3 14B Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
said the passages give no difference. Its two quotes are altered. They start with the passage number and the term and leave out the lesson name, which the passage holds. The words of the entry itself match.
You sent:Passages:
[1] epoch (from the lesson "Training"): Google's word for one round, one pass over all the examples.
[2] round (from the lesson "Training"): One pass over all the examples. After each round, training changes the model's numbers a little.
[3] training (from the lesson "Training"): Changing a model's numbers, round after round, so its guesses on the examples get less wrong.
Question: What is the difference between a round and an epoch in training?
The passages do not provide a clear difference between a round and an epoch in training. Both terms are used interchangeably in the given passages:
"[1] epoch: Google's word for one round, one pass over all the examples."
"[2] round: One pass over all the examples."
Question 5: How many tokens fit in the context window of Claude Sonnet 5?
Claude Sonnet 5.5 Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
said the passages give no figure. Every quote matches.
You sent:Passages:
[1] context window (from the lesson "Attention and the context window"): The most text a model can take in at once, counted in tokens; it includes the model's answer.
[2] model (from the lesson "Bad and good prompts"): The AI system that writes the answer. This lesson uses two, Claude Sonnet 5.5 and Ministral 3 14B.
[3] token (from the lesson "Examples"): A small piece of text that an AI reads, such as a word, part of a word or a letter.
Question: How many tokens fit in the context window of Claude Sonnet 5?
The passages do not say how many tokens fit in the context window of Claude Sonnet 5. Passage [1] defines the context window only in general terms, as "The most text a model can take in at once, counted in tokens; it includes the model's answer." [1] Passage [2] mentions "Claude Sonnet 5.5" as one of two models used in a lesson, but it gives no token limit. [2] Passage [3] defines a token as "A small piece of text that an AI reads, such as a word, part of a word or a letter." [3] None of them gives a figure, and the version named in [2] is 5.5 rather than 5, so I can't answer the question from these passages.
Ministral 3 14B Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
said the passages give no figure.
You sent:Passages:
[1] context window (from the lesson "Attention and the context window"): The most text a model can take in at once, counted in tokens; it includes the model's answer.
[2] model (from the lesson "Bad and good prompts"): The AI system that writes the answer. This lesson uses two, Claude Sonnet 5.5 and Ministral 3 14B.
[3] token (from the lesson "Examples"): A small piece of text that an AI reads, such as a word, part of a word or a letter.
Question: How many tokens fit in the context window of Claude Sonnet 5?
The passages provided do not answer the question about how many tokens fit in the context window of Claude Sonnet 5. They do not mention this specific model's context window size.
Question 6: How much of a page's text does a citation show?
Claude Sonnet 5.5 Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
gave the limit for the one tool the passage names, and said the passages give no other amount. Every quote matches.
You sent:Passages:
[1] citation (from the lesson "Where your question goes"): A note in an answer that points to a page. In Anthropic's web search tool, it gives the page's address, title and up to 150 characters of its text.
[2] citation (from the lesson "Why they make things up"): A reference to a source, such as a study, so a reader can find it. It gives details like the authors, year and journal.
[3] context window (from the lesson "Attention and the context window"): The most text a model can take in at once, counted in tokens; it includes the model's answer.
Question: How much of a page's text does a citation show?
In Anthropic's web search tool, a citation gives the page's address, title and "up to 150 characters of its text" [1]. The passages do not say how much text a citation shows in any other setting. Passage [2] describes citations to sources such as studies, but it only lists details like authors, year and journal and does not mention any amount of text [2].
Ministral 3 14B Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
gave the limit for the one tool the passage names. The quote matches.
You sent:Passages:
[1] citation (from the lesson "Where your question goes"): A note in an answer that points to a page. In Anthropic's web search tool, it gives the page's address, title and up to 150 characters of its text.
[2] citation (from the lesson "Why they make things up"): A reference to a source, such as a study, so a reader can find it. It gives details like the authors, year and journal.
[3] context window (from the lesson "Attention and the context window"): The most text a model can take in at once, counted in tokens; it includes the model's answer.
Question: How much of a page's text does a citation show?
A citation in Anthropic's web search tool shows "up to 150 characters of its text" [1].
Question 7: Under the AI Act, who counts as a deployer?
Claude Sonnet 5.5 Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
kept the exception for personal use. Every quote matches.
You sent:Passages:
[1] deployer (from the lesson "The law: AI rules in Ireland and the EU"): In the AI Act, whoever uses an AI system under its authority. Personal, non-professional use is left out.
[2] provider (from the lesson "The law: AI rules in Ireland and the EU"): In the AI Act, whoever develops an AI system, or has one developed. It puts the system on the market under its own name.
[3] high-risk AI system (from the lesson "The law: AI rules in Ireland and the EU"): An AI system the AI Act puts under extra rules. The uses on its Annex III list are high-risk.
Question: Under the AI Act, who counts as a deployer?
Under the AI Act, a deployer is "whoever uses an AI system under its authority." [1] Personal, non-professional use is left out of this definition. [1]
Ministral 3 14B Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
one claim goes beyond the passage. It adds that deployers are typically those using AI systems in a professional or organizational capacity. The passage says only that personal, non-professional use is left out. We do not call the added claim false, only unsupported by the passage.
You sent:Passages:
[1] deployer (from the lesson "The law: AI rules in Ireland and the EU"): In the AI Act, whoever uses an AI system under its authority. Personal, non-professional use is left out.
[2] provider (from the lesson "The law: AI rules in Ireland and the EU"): In the AI Act, whoever develops an AI system, or has one developed. It puts the system on the market under its own name.
[3] high-risk AI system (from the lesson "The law: AI rules in Ireland and the EU"): An AI system the AI Act puts under extra rules. The uses on its Annex III list are high-risk.
Question: Under the AI Act, who counts as a deployer?
Under the AI Act, a deployer is defined as "whoever uses an AI system under its authority." [1]
The passage also clarifies that "Personal, non-professional use is left out," meaning deployers are typically those using AI systems in a professional or organizational capacity. [1]
Question 8: What is fine-tuning?
Claude Sonnet 5.5 Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
said the passages do not define the term. It put "[1]" and "[2]" after sentences that cover all three passages.
You sent:Passages:
[1] attention (from the lesson "Attention and the context window"): In a language model, the weighing of earlier pieces of text, done with numbers, to write the next piece. The word is borrowed from people.
[2] regulation (from the lesson "The law: AI rules in Ireland and the EU"): An EU law that is binding in full and applies directly in every EU country.
[3] made-up detail (from the lesson "Why they make things up"): A detail in an answer that sounds real and is not. It could be a study that does not exist, a wrong year, or a name that does not belong.
Question: What is fine-tuning?
The passages do not define fine-tuning, so I can't answer the question from them. [1] They cover only "attention," "regulation," and "made-up detail," and none of those entries mentions fine-tuning. [2]
Ministral 3 14B Run 1 Recorded 8 October 2026
An instruction we put behind the message: "Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage's own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else."
said the passages do not answer.
You sent:Passages:
[1] attention (from the lesson "Attention and the context window"): In a language model, the weighing of earlier pieces of text, done with numbers, to write the next piece. The word is borrowed from people.
[2] regulation (from the lesson "The law: AI rules in Ireland and the EU"): An EU law that is binding in full and applies directly in every EU country.
[3] made-up detail (from the lesson "Why they make things up"): A detail in an answer that sounds real and is not. It could be a study that does not exist, a wrong year, or a name that does not belong.
Question: What is fine-tuning?
The passages provided do not answer the question about fine-tuning.
A close match is not a right answer. Check the answer against the passage it points to.
Lists made 9 October 2026 with nomic-embed-text, for a snapshot of the glossary: 159 entries from the first 25 lessons. Nothing is counted or saved.
Step two: the answer
The second step is the answer. We sent each of the eight questions to two models, with the top three passages our own code found. They were Claude Sonnet 5.5, a model from Anthropic, and Ministral 3 14B, a model from Mistral AI.
Each got each question once, on 8 October 2026.
We told both the same rules. Use only the passages. Put the passage number after each sentence. Quote the passage’s own words at least once. If the passages do not answer, say so.
The models did not search. They saw only what we sent. We then checked every sentence of all sixteen answers against the three passages.
Our system prompt was: “Answer the question using only the passages. After each sentence of your answer, give the number of the passage it comes from in square brackets, like [2]. Quote the passage’s own words in double quotation marks at least once. If the passages do not answer the question, say so, and do not answer from anything else.” It is a plain, strict instruction. We did not soften it to make mistakes more likely.
We ran each question once per model. Neither model had tools. We did not set the temperature, so the services used their own default value. Every run is on the page, with its model, date and settings.
What sixteen answers show
Claude Sonnet 5.5 stayed inside the passages in all eight answers. Every quote matched the passage word for word.
Ministral 3 14B stayed inside in four answers. In four it did not.
Question 1, what a token is. A token is a small piece of text, often part of a word, that a language model works with. Ministral added a comma before “or a letter” in a quote. The passage has no comma there.
Question 3, tag characters. Both models said Unicode has 96 tag characters. Unicode counts 97. The passage’s 96 are a range of them, not all. The passage says only the last one ends a tag. Ministral wrote “end a tag”, and said they are used to end a tag.
Question 4, round and epoch. Ministral’s two quotes begin with the passage number and term, such as “[1] epoch:”. They leave out the lesson name that the passage holds.
Question 7, deployers. In the AI Act, a deployer is whoever uses an AI system under its authority. Ministral added that deployers are “typically” people using AI at work. The passage says only that personal, non-professional use is left out.
Staying inside the passages is not the same as being true. Our passages are short glossary entries, and they leave things out. An answer built from a store can repeat the store’s gaps.
Questions 2, 5 and 8 had passages that did not hold the answer. All six answers said so. We had told both models to say that when the passages did not hold the answer.
This is eight questions, two models and one day. It does not show how often any system misquotes.
We counted two things. A quote is altered when a word or punctuation mark inside the quote marks is added, left out or changed. A statement goes beyond the passages when the passages do not say it, even if it is true. We ignored the quote marks, spacing and a capital letter at the start of a quote. We also ignored a comma or full stop placed just inside the closing quote mark.
By that rule, four quotes were altered, all in Ministral 3 14B answers. Ministral also made two statements that go beyond the passages. One says tag characters are used to “end a tag”. The other says deployers are “typically those using AI systems in a professional or organizational capacity”. Two of the four altered quotes are in one answer, to question 4. Another sits in an answer that also goes beyond its passage. In all, four of sixteen answers changed a quote or went beyond the passages: none of Claude’s eight, four of Ministral’s eight.
Both answers to question 3 say Unicode has 96 tag characters. The Unicode Standard says its Tags block “encodes a set of 97 special-use tag characters”. One of the 97, U+E0001, is deprecated, which Unicode defines as a character “whose use is strongly discouraged”. The 96 in our passage run from U+E0020 to U+E007F. A passage can support an answer and still be incomplete. The Unicode Standard does not say that all of them are used to end a tag. It says they are encoded “to enable the construction of tags”.
In Ministral’s answer to question 4, the quotes begin with the passage’s number and term, such as “[1] epoch:”. They leave out the lesson name that the passage holds. The words of the entry itself match, but under our rule the quotes are altered, so we counted them.
Claude’s answer to the fine-tuning question put “[1]” after a sentence about all three passages, and “[2]” after a sentence that names all three. A single passage does not cover either sentence. We marked this as a note.
Two of the passages for question 7 leave something out. Article 3(3) of the AI Act counts as a provider a body that “places it on the market or puts the AI system into service under its own name or trademark”. Our provider entry names only the market. Article 6(3) says an Annex III system “shall not be considered to be high-risk” where it “does not pose a significant risk of harm”. It lists conditions where this applies, such as performing “a narrow procedural task”. Our high-risk entry says the Annex III uses “are high-risk”.
The round entry says “After each round, training changes the model’s numbers a little.” Google’s glossary says an epoch is a full pass over the training set. It says an epoch “represents N/batch size training iterations”, and an iteration is “A single update of a model’s parameters”. So the numbers can change many times in one epoch.
Question 2 shows a different limit. Both models said the three passages did not answer. That was right for those passages. It does not mean the snapshot lacks the answer. The retrieval step had missed it.
A study of tools that find first
Do real products that find first still make mistakes? A study run in 2024 asked this about three legal research tools. These tools help with tasks such as searching case law, the opinions that judges have written.
The researchers asked each tool the same 202 legal questions. People checked every answer by hand. They counted an answer as bad if it had a false statement. They also counted it if it said a source backs a claim when it does not.
Lexis+ AI was bad in 17% of answers, about 1 in 6. Ask Practical Law AI was bad in 17%, about 1 in 6. Westlaw AI-Assisted Research was bad in 33%, about 1 in 3.
The study ran from March to May 2024. It describes those versions on those dates. It does not describe the products today.
The three tools come from two companies. LexisNexis makes Lexis+ AI. Thomson Reuters makes Westlaw AI-Assisted Research and Ask Practical Law AI. Queries on Lexis+ AI, Ask Practical Law AI and GPT-4 ran from 22 March to 22 April 2024. Queries on Westlaw ran from 23 to 27 May 2024. The paper appeared in the Journal of Empirical Legal Studies on 23 April 2025.
The authors define a hallucination as “a response that contains either incorrect information or a false assertion that a source supports a proposition”. GPT-4, from OpenAI, is a general chatbot model. The authors asked it with no retrieval step. It was bad in 43% of answers, about 4 in 10. The authors write that hallucinations were “reduced relative to general-purpose chatbots (GPT-4)”. The three tools still made them. The authors also write: “Because these are closed systems, we are not able to clearly identify a single point of failure for each hallucination.” The authors report that the companies’ claims of “hallucination-free” results were overstated.
The study also uses a word for our topic. An answer is “misgrounded” when its key claims are cited but “misinterpret the source or reference an inapplicable source”. The authors note that their use of “grounded” “deviates somewhat from the notion in computer science”. We use a plainer test: does the answer stay inside the passages?
How to check an answer against its passages
These habits come from the sixteen answers and the study.
Read the passage beside the answer. A number after a sentence only says where the model claims it came from.
Compare quotes word for word. One missing letter changed “ends a tag” into “end a tag”.
Look for what was added. “Typically” was not in the passage.
Look for limits that were dropped. One passage said personal use is left out.
Check whose rule it is. In question 6, one passage gave a limit for one tool only. Both answers said so.
A close match, a passage number and a plain instruction did not stop mistakes in our test. Comparing the passages with the answers is what revealed them.
“I can’t answer from these passages” can be the right reply. It does not mean the answer is missing from your documents. In question 2 it only meant that the retrieval step had missed it.
The study found that systems can also cite a source that does not support the claim. So a citation is a pointer, not a proof. Open the passage and find the words.
Lesson 4.1, Check it before you use it, covers checking an answer against real sources. Lesson 5.6, Prompt, RAG or fine-tune?, compares RAG with two other ways to shape an AI.
Check-in
In this lesson's activity, what does a "close" match tell you?
The question and the passage are alike, by one model's lists of numbers. You still read the passage.
Yes. Similarity compares two lists of numbers. It does not show that the passage answers the question.
The passage answers the question.
Not this one. For a question the snapshot does not cover, the system still returned five entries. None of them was close.
The passage is true.
Not this one. The number compares two lists of numbers. It does not check whether a passage is true.
You did well.
Not this one. The similarity is about a question and a passage. It says nothing about you.
A model was given three passages and wrote an answer that quotes one of them. What is the safest next step?
Compare the quote, and each claim, with the passage's own words.
Yes. In our test, one answer changed "ends a tag" to "end a tag". Another added a claim the passage did not make.
Trust it, because a passage number follows each sentence.
Not this one. A number says where the model says a claim came from. The study counted answers that cite a source which does not support the claim as bad.
Trust it, because the three passages were very close matches.
Not this one. For our first question, all three passages were very close. One answer still changed a quote.
Trust it, because the model was told to use only the passages.
Not this one. We told both models that. Four of the sixteen answers still changed a quote or added a claim.
Our recording R-5.5, 8 October 2026, and our glossary vectors, computed 9 October 2026 (data/recorded/R-5.5.json, data/interactive/glossary-vectors.json) (not published) (All rights reserved, cited with credit)
Glossary
RAG
Retrieval-augmented generation. A system first finds passages in a store of text, then a language model writes an answer using them.
token
A small piece of text, often part of a word, that a language model works with.
chunk
A short piece cut from a longer text. The 2020 RAG paper cut each Wikipedia article into chunks of 100 words.
passage
A short piece of text that a system finds for a question. In this lesson, a passage is one chunk.
embedding
A list of numbers that stands for a piece of text. Pieces that are alike should get lists that sit close together.
similarity
How close two embeddings are. Here it is the cosine of the angle between them. It does not show that a passage is true or answers the question.
training data
The large body of text that a language model is trained on.
temperature
A setting that controls how random a model's output is.
context window
The amount of text a model can look back on when it writes.
Where next
The 2020 paper that used the name RAG. arXiv is a website where researchers share papers. The page has the summary and a link to the full paper. NLP in the title means natural language processing, the handling of human-language information by a computer. It is written for researchers. The page loads files from arxiv.org, static.arxiv.org, code.jquery.com and cdn.jsdelivr.net. A cookie is a small piece of data that a site sends to your web browser. It sets one cookie, named arxiv_labs, before any choice. We saw no other hosts. We checked this on 9 October 2026.
