Most Documented AI Failures Were Found by Someone Else

Who found the failure

A record can name more than one finder, so the percentages add to more than 100%.

Even the inside catches came late

In all but one of the records that name the organization’s own people among the finders, the output had already been acted on by the time the problem was found: published to readers, given to customers, used to buy homes or make an arrest, paid out, or put on the road. In the exception, a school system’s staff canceled a false AI gun alert, but others who had received it carried on, and police searched the school anyway. Three show the pattern:

• The money had already gone. At Arup, in a case reported in February 2024, an employee in the Hong Kong office joined a video call in which every other participant was a deepfake, then made 15 transfers totaling HK$200 million to five bank accounts. The fraud surfaced only when the employee later checked with head office.
• The homes had already been bought. In its third-quarter 2021 results, published in November 2021, Zillow recorded an inventory write-down of about $304 million because it had bought homes at prices above its own current estimates of what they would sell for.
• The arrest had already been made. A Detroit detective realized that Porcha Woodruff could not be the carjacking suspect, because she was eight months pregnant, only on seeing her at the detention center after her arrest.

That is a finding about public failures, not about all failures. Errors that an organization stops on its own are less likely to become public, so a record built from public sources will over-represent outside discovery, and the 16% does not measure how often organizations catch their own mistakes. What it does show is where detection happened in the failures that became public: after the output had been relied on, and usually by someone with no reason to contain it.

Finding 1: The most common failure was the easiest to check

Fabricated content is the largest category: in 38% of the incidents, invented citations, references, quotations or figures were relied on or published. Of those, 43% were court filings in which the court or the opposing party found the problem. In Mata v. Avianca, six of the decisions cited to a federal court in New York appeared to be bogus; in Wadsworth v. Walmart, eight of nine cited cases did not exist. Another 24% were reports published by large professional-services firms, discussed in When a Report Cites Sources That Do Not Exist.

Each of these errors was a reference someone could have looked up before the document left the organization. That is why a check of output against its source is the safeguard the lessons name most often: 37% of the reviewed lessons. The same failure can appear wherever AI drafts text that cites something: a policy paper, a news story, a grant application, a court filing or a client report.

Finding 2: A review was in place, but it could not have caught the error

Several failures passed a person whose job was to check. In Wadsworth, the court noted that two of the three lawyers who signed the motions had not reviewed them before signing, and on February 24, 2025 fined each of them $1,000. At Cigna, according to company documents reported by ProPublica and The Capitol Forum, company doctors denied more than 300,000 requests for payment over two months in 2022, spending an average of 1.2 seconds on each; Cigna called the reporting biased and incomplete. The reporting does not show that any particular denial was wrong; it shows a review too fast to find out. When G/O Media published an AI-written article on its io9 site, the site’s deputy editor said he had been told about ten minutes beforehand and that no one at io9 had edited it.

The lessons name a review that could not catch the error in 17% of cases. A sign-off shows that someone approved the work. It does not show that the person had the time, the information or the expertise to find the mistake.

Finding 3: Responsibility did not move to the tool or the contractor

Using a chatbot, a contractor or an AI tool did not move the consequence away from the organization whose name was on the output. A British Columbia tribunal found Air Canada liable for negligent misrepresentation because it did not take reasonable care to ensure that the information its website chatbot gave about bereavement fares was accurate. The Court of Appeal of Alberta held that the lawyer whose name appears on a filed document bears ultimate responsibility for its contents, even where a third-party contractor prepared it, according to ABlawg’s account of the decision, and in January 2026 ordered the appellant’s lead counsel to pay C$17,550 plus GST in costs personally. Deloitte agreed to repay the final instalment of its contract with Australia’s Department of Employment and Workplace Relations after the review it delivered was found to contain errors, and amended the report to disclose that part of the work used a generative AI tool.

Whoever relies on AI output owns its errors, whether the AI was its own, built into a supplier’s product, or used by an adviser.

Finding 4: Some of the largest harms came from automated decisions that were hard to challenge

Some of the largest documented harms in the record come from systems that are not AI. According to the Royal Commission’s report of July 2023, Australia’s Robodebt scheme ended with about A$746 million repaid to some 381,000 people and A$1.751 billion in debts written off. More than 900 UK sub-postmasters were wrongly prosecuted because of incorrect information from the Post Office’s Horizon computer system. In both, the system’s output was treated as proof: the Royal Commission described the reversal of the onus onto the recipient as one of Robodebt’s fundamental features, and sub-postmasters were prosecuted on the Horizon system’s incorrect information. The lessons for these cases, and for Idaho’s Medicaid budget tool, name the absence of a route to challenge; When the System Is Presumed Right sets out the pattern and the safeguards.

Finding 5: A claim about AI can itself be the failure

Three incidents in the record concern what organizations said their AI could do rather than what it did: the SEC’s orders against two investment advisers, the FTC’s case against Air AI and its order against Workado. In March 2024 the SEC settled charges against Delphia and Global Predictions over false and misleading statements about their purported use of AI; without admitting or denying the SEC’s findings, they agreed to pay civil penalties of $225,000 and $175,000. When the Claim About AI Is the Failure follows the enforcement since.

What would have prevented them

The table ranks safeguards by how often the reviewed lessons name them as missing or failed. The lessons are our analysis of each incident, reviewed in a separate pass before publication; they are not findings of the courts or regulators involved.

The ranking reflects which failures became public, not which safeguards matter most in a given organization. A safeguard named once in the record, such as confirming a payment request through a separate channel, can still be the one that matters most where money moves.

Are you at risk?

The questions follow the safeguards above, most often missing first. Each “no” is a gap the record shows others found the hard way.

1. Is everything your AI produces checked against its source before it reaches a reader, customer, court or decision?
2. Was the tool tested on cases with known answers, including unusual ones, before it was used on live work?
3. Do you track errors, overrides and complaints, with a set level that triggers investigation?
4. Do you know that the data the tool received was complete and current?
5. Does the person who signs off have the source, the time and the expertise to catch the error, and is what they checked recorded?
6. Is a named person accountable for each AI-assisted output, and do the people who receive it know AI was used?
7. Do your suppliers and advisers tell you where they use AI in what they deliver, and how they check it?
8. Is the tool prevented from publishing, committing, paying or deciding beyond limits a person set?
9. Can someone affected by an automated outcome challenge it and reach a person who can change it?

What to do next

1. List where AI output is relied on without a check against its source. Start with what leaves the organization: published material, filings, client deliverables and decisions about people.
2. Make the source check part of preparing the work, not part of approving it. The person who prepares the work attaches the trace; the reviewer confirms it.
3. Ask suppliers and advisers whether AI is used in what they deliver and how its references and figures are checked.
4. Give people a route to challenge automated outcomes that affect them, and treat disputes as monitoring data.
5. Use the record. Each incident, with its sources and its lesson, is in the failures section of Eminence Scan, which can be filtered by the safeguard that was missing.