Where the answer comes from

Where the answer comes from

1. A person asks an assistant to briefly summarize a memo about an office lease and within seconds gets three even sentences: the parties and the term, rent of UAH 35,000 per month, a penalty of 0.1% a day. The model took the first from the memo, the amount in the second belongs to something else, and the third is not in the memo at all. They look no different. To know what to check, trace where the model takes each statement from.

2. The first source of statements is the text the model receives together with the request. The model itself does not keep the conversation, so each time the product assembles this text anew: its own instructions, the attached memo, the conversation history and the latest message. All of it together is called the context. Here it holds two amounts: UAH 30,000 in the memo and UAH 35,000 in an earlier message about the office next door.

3. From the context the model builds the answer not whole but one word or part of a word at a time, each time continuing what is already written. For every next word it computes a probability, a number from zero to one that shows how well the word fits after the previous ones. After “from 1 February 2026 for 12” nearly all the probability goes to “months”, because that is what the memo says. The numbers in the film are illustrative.

4. The model picks a word by these probabilities, but with a share of randomness, so sometimes it takes not the likeliest one. After “The rent is UAH” both amounts from the context have a noticeable probability: 30,000 from the memo a larger one, 35,000 from the conversation a smaller one. This time the model picked 35,000, and the sentence is then built around it. The mechanism itself has no step at which the amount would be checked against the memo.

5. That is how statements from three different sources end up side by side in the finished answer. The model took the parties and the term from the attached material, the memo. The amount of UAH 35,000 came from the conversation: the person wrote it, only about another office. The penalty comes from training: before release the model was trained on a vast amount of text, and what remains of it is mainly patterns of which words usually follow which.

6. The third source is the least visible, and hallucinations come from it. The memo says nothing about a penalty, but in the leases the model met in training a penalty usually follows the rent, quite often exactly 0.1% a day. So that continuation has a high probability, while the words “the memo does not say” are rare in texts. A statement with the right form but no source is called a hallucination.

7. By tone alone a hallucination cannot be told from a checked statement. The model has internal signals of whether a statement is familiar to it, but they are unreliable and invisible in the answer. A confident tone is also assembled word by word as the most usual continuation, so it is only loosely tied to correctness. All three sentences sound equally even, although the memo confirms only one.

8. Since the tone gives no clue, the answer is split into separate statements, and each gets one question: where is this written in the material. This answer has three statements: who rents from whom and for how long, what the rent is, and what the penalty is. The concrete ones are checked first, that is names, dates, amounts, percentages and quotations: hallucinations are frequent there, and a mistake costs a lot.

9. The check itself is simple: the answer is placed next to the memo, and for each statement you look for the line that confirms it. The parties and the term match point 1. In point 2 the rent is UAH 30,000, not 35,000, so the amount is corrected. The memo has no point about a penalty, so that statement stays without a source and is removed from the summary. The memo confirms one of three.

10. Asking again does not replace this check. The model picks words with a share of randomness, so the same request can produce different text: here the second answer already has rent of UAH 30,000 and no penalty. If the answers had matched, that would prove nothing, because a repeat compares them with each other, not with the memo. A difference is still a useful hint: the places where answers differ are checked first.