A Shorter Version Must Not Become a Stronger Claim

Summaries widen claims, and asking for accuracy did not help

A study published in Royal Society Open Science in April 2025 compared 4,900 summaries from ten language models with the scientific texts they summarized. Most models produced broader generalizations than the originals, even when told to be accurate; for three of them the study’s overall overgeneralization measure ranged from 26% to 73%. On 100 medical articles, the odds of a generalized conclusion in summaries by three of the models were 4.85 times those in summaries written by medical experts. That is an odds ratio, not a fivefold probability. A prompt asking the models to avoid inaccuracies was associated with about twice the odds of generalization compared with the simple prompt. Newer models in that comparison tended to do worse than older ones.

The most common widening was that a quantified claim, such as “75% of” a group, became a general statement about the whole group. That change is small and easy to miss. The topic survives compression. The scope often does not.

Quotations become paraphrase, and paraphrase becomes fact

In the European Broadcasting Union and BBC’s 2025 study of the free versions of four AI assistants, journalists from 22 public broadcasters checked 2,709 answers about the news. Of the answers that contained a direct quotation, 12% had significant problems with it. The report gives examples of quotes invented outright and of quotes reworded while still presented as quotes. One evaluator found an assistant “takes quotes from the interviewee and transforms them into facts”.

Headlines are the shortest summaries of all. In December 2024 and January 2025, Apple’s AI summaries of news notifications falsely told readers that a murder suspect had shot himself and that a darts player had won a championship that had not yet started; the BBC said the summaries in some cases completely contradicted its reporting. The summaries appeared as if they came from the news organization’s own app. In January 2025 Apple disabled the summaries for news and entertainment apps in a test version of its next iPhone software while it worked on the problem.

Two weeks after the retraction, Ars Technica said it would publish a reader-facing guide to how it uses and does not use AI. That policy page, last updated April 22, 2026, says AI tools must not be used to generate, extract or summarize material that is then attributed to a named source, whether as a quotation, a paraphrase or a characterization of someone’s views.

A correct number can tell the wrong story

Charts and headline figures compress too. In July 2015 an automated Associated Press earnings story said Netflix shares had “decreased 71% since the beginning of the year”. The shares had in fact doubled that year; in July 2015 Netflix had just split its stock seven for one, so the price per share fell from about $700 to about $98 with no change in what holders owned. The subtraction was not the error; the comparison was: a post-split price set against a pre-split one.

The same thing happens with denominators. In an invented example, 1,000 people are invited to a survey, 400 respond and 280 of them are favorable. That is 70% of respondents and 28% of the people asked; the 600 who did not answer are unknown, not opposed. A chart titled “70% of employees agree” has changed the population. In another, a pilot cuts turnaround from 10 days to 8, a 20% reduction. A slide headline reading “AI delivers a sustained 20% productivity gain across the business” adds three claims the pilot did not test: sustained, productivity and across the business.

For finance reporting, the release standard for charts is set out in A Chart Is a Claim.

How to check a shorter version

1. Check every quotation against the source text, word for word, and the speaker against the source. If the AI extracted it, treat it as unverified until someone has found it in the original.
2. Compare the strength of each consequential sentence with the source: “may” against “will”, “some” against “all”, “associated with” against “caused”, one case against a pattern.
3. Check the population, period and denominator behind every percentage and chart. Write the numerator and denominator down without looking at the title, then see whether the title says the same thing.
4. Read the headline and slide titles last, as claims. Each scope word (sustained, across, all, every) needs evidence of its own.
5. Keep the limitation next to the claim it limits, not in a footnote the headline contradicts.
6. Inspect what the audience will actually see: the rendered slide, the exported file, the notification on the phone.

Where this applies across the centers

• Finance: board summaries of variance analysis; a chart of an adjusted measure without the reported one.
• People teams: survey results reported as the view of “employees” when a minority responded; interview notes summarized into a candidate assessment.
• Buying AI: vendor case studies summarized into “customers see 30% savings”; check which customers, measured how.

Are you at risk?

• Do AI-drafted documents that quote people go out without each quote being found in its source?
• Do executive summaries keep the sample size and the exceptions that would change the decision?
• Are chart titles checked against the numerator and denominator behind them?
• Is the version reviewed the version that is sent, or is it edited again after review?

Release the summary only after checking its scope

Making something shorter is a judgment about what can be left out. Before release, put the short version next to its source and check the quotes, the scope words and the denominators. Those are what goes missing first.