Skip to content

ResearchNote

When true fragments add up to a false whole

You can gather facts that are each correct and still end up with the wrong picture. The mistake lies not in the fragments but in how they are joined. Using exam results and book sales as examples, this note separates two ways it happens.

  • Knowledge
  • Agents

Suppose a figure like this appears for an exam:

  • 80% of the people who passed were students at Prep School A.

It seems to say that attending A makes you more likely to pass. But this figure alone can’t tell you that.

What the number counts

The 80% is measured against the people who passed. It tells you how many of those who passed were A’s students. To compare whether A’s students pass more often, you need the figure the other way around: what share of A’s students passed, their pass rate. In statistics, this is known as confusing a conditional probability with its inverse.

Say 1,000 people took the exam, and 900 of them were A’s students. Of the 200 who passed, 160 were from A. So 80% of those who passed were indeed A’s students. But A’s pass rate was 160 out of 900, about 18%, while everyone else passed at 40 out of 100, or 40%. Among these test takers, A’s students had the lower pass rate.

Even so, this doesn’t tell you what attending A does. The two groups may have differed before they started, in their earlier grades or in their reasons for taking the exam. Comparing pass rates still leaves that condition out.

Even if a figure is correct, mistaking what it was measured against, or the conditions behind it, builds a false picture. Correct numbers alone aren’t enough. You need to check who each number was measured against, when, and under what conditions, and you need a model that ties the numbers together.

Two ways to a false whole

Generalizing from this example, there are at least two ways correct fragments can add up to a false whole. They can happen separately or together.

The relationships go missing. Facts sit apart from one another, with no model connecting each to the conditions under which it holds. In the exam example, the share of passers who came from A isn’t separated from A’s pass rate, and the differences between the two groups go unexamined, so the link between the numbers and their conditions is lost.

The surface is mistaken for the mechanism. An observed result or figure is treated as an explanation of what produced it. Say a book sells well. Its sales show that many people bought it. What produced that result isn’t only the content: timing, price, publicity, and luck all play a part. And even if the content drew readers in, being appealing is not the same as being right. Read the sales as proof that the ideas in the book are correct, and you’ve mistaken a resulting figure for the mechanism behind it. Whether the ideas are correct has to be checked separately from the sales.

Checking against reality

When the two overlap, the problem grows. If a picture built from a limited set of fragments is treated as an explanation of the whole, and only the fragments that fit it keep being added, it can look coherent on the inside while the gap with reality stays or widens.

Finding the gap means working out what should happen if the explanation is right, checking that through observation or comparison, and revisiting the explanation when the results don’t fit. If results that don’t fit never lead to a revision, the explanation holds together only on its own terms, and reality can no longer correct it.

None of this is a rejection of science, expertise, or large models. Any model can fall into this failure once it loses contact with reality, and the ideas in this note are no exception.

Reading an AI’s answer

Large language models are very good at producing fluent text from fragments. But fluent text is not the same as relationships between fragments that actually hold. The two mistakes above can happen when reading an AI’s answer, too.

For example, suppose you ask an AI how a relationship begins in which two people read and adjust to each other’s states, and it answers with terms borrowed from neighboring fields: causality, observation, feedback. It reads like an explanation. But listing terms doesn’t say what sets things off, who changes and how, or how the other person responds. Those terms become an explanation only once that content is spelled out.

Names and explanations

Naming what is happening, listing the fields it touches, and explaining how it happens are three different things.

Names, categories, provisional frameworks, and models are tools that make a question easier to handle and make the next comparison possible. But being useful is not the same as explaining a phenomenon correctly. Naming something doesn’t mean the mechanism behind it is understood. Rather than being fixed as final answers, these tools get rebuilt as understanding and observation improve.

What to do

  • When using a number, check and record its source and the date you looked it up, along with what it was measured against, when, and under what conditions.
  • When comparing two groups, check whether they already differed before the comparison.
  • Write down confirmed facts, the explanations drawn from them, and hypotheses not yet checked, separately.
  • Look for cases that don’t fit your explanation, and for other explanations.

Open questions

  • Beyond whether the fragments are correct, how do you check whether the way they are joined is correct?
  • Is there a way for writers to catch themselves mistaking an observed result for the mechanism behind it?
  • What would a reader need to tell which parts of an explanation are confirmed relationships, and which are hypotheses or rephrasing?
  • When does a name or framework go beyond organizing a phenomenon and make new observations possible?

Version history

  1. v0.1Oct 7, 2026First version.

This document’s URL will not change. When citing it, please include the version number.

All research