When the System Is Presumed Right

How long the output was treated as right

The bars measure how long a system’s output stood before a court, a law or the operator itself accepted that it was wrong. The contrast is the point: where the people affected could not challenge the output, errors stood for years; where complaints reached the operator quickly and it acted on them, a flawed check was stopped within weeks.

Post Office Horizon: the presumption written into law

Between 2000 and autumn 2013 the Post Office prosecuted postmasters in England and Wales on the strength of Horizon accounting data. Postmasters complained that Horizon produced false data almost from the start, and the Post Office trenchantly resisted those complaints for the life of the first two versions of the system. The inquiry chair found that a number of Post Office employees knew, or should have known, that Horizon could be wrong, yet the Post Office maintained the fiction that its data was always accurate. He found it likely that about 1,000 people were prosecuted and convicted across the United Kingdom on Horizon evidence, while thousands more were suspended over apparent shortfalls.

It took postmasters themselves to break the presumption: about 555 of them brought group litigation that ended after the court accepted their main contentions. Parliament quashed the convictions by statute in May 2024. By September 2026 the Ministry of Justice had identified 620 people in England and Wales whose convictions the Act quashed, and by September 30, 2026 about £1,731 million had been paid in redress to more than 14,000 claimants. Only the first volume of the inquiry’s final report, on redress, has been published; in July 2026 the chair said the rest was still some months away.

The legal presumption behind it still stands. Since 2000, English criminal law has presumed that a computer was operating correctly unless shown otherwise; a courts minister has described it as “the computer is always right”. The Ministry of Justice consulted on reform between January and April 2025, proposing that any change cover evidence from software including AI and algorithms, and has not yet published an outcome. It attributed the Post Office miscarriages to deliberate failures to interrogate and disclose evidence, which stopped postmasters from challenging Horizon.

Robodebt: the onus reversed

Australia’s Robodebt scheme raised welfare debts by averaging income, and the Royal Commission described the reversal of the onus onto the recipient, together with income averaging, as its fundamental features. The scheme ran from about July 2015 to November 2019. Approving a further settlement in June 2026, worth up to A$548.5 million on top of the A$112 million paid under the 2021 settlement, the Federal Court recorded that the Commonwealth unlawfully asserted debts of at least A$1.763 billion against about 433,000 people.

The error was found early and ignored. The welfare peak body set out many of the scheme’s flaws in a December 2016 letter to the minister. A tribunal ruled income averaging unlawful in March 2017, but the department did not appeal or share that decision and similar ones. The Royal Commission found that a system to review those decisions would have shown rejections of averaged debts far outnumbering acceptances. The Commonwealth first conceded in court that income averaging was unlawful in November 2019. In March 2026 the National Anti-Corruption Commission found that two of the six people referred by the Royal Commission had engaged in serious corrupt conduct, and referred no one for prosecution, citing insufficient admissible evidence.

Idaho, Michigan and a 2025 benefit check

Idaho. When Idaho’s Medicaid budget tool cut disability support, the state at first declined to disclose the formula, calling it a trade secret, according to The Verge. The federal court found that the tool’s unreliability arbitrarily deprived participants of their property rights and violated due process, and that before the state removed human reconsideration, 62% of reconsidered budgets had been increased. As of August 2026 the replacement budget model was still not in use.

Michigan. From October 2013 Michigan’s MiDAS system started an automated fraud process that allowed no fact-based adjudication. A state appeals court, quoting a federal court, said a review of more than 22,000 of its fraud determinations found 93% did not involve fraud. The state amended its law in 2017, effective July 2018, to let claimants reopen fraud findings, and a US$20 million class settlement was approved in January 2024.

The UK’s 2025 Child Benefit check. From August 2025 HM Revenue & Customs matched Child Benefit records against Home Office travel data, and by October it had opened enquiries on 23,794 claimants, suspending their payments immediately and asking them for evidence that they lived in the UK. For the rollout it had removed its main upfront employment check, and the Home Office’s data matching used five years of travel data instead of the 12 months used in the pilot. By mid-October 2025 it was clear many eligible claimants had been suspended, including people who had simply returned to Northern Ireland through Ireland. The National Audit Office estimated that 59% of the cases were confirmed eligible or removed after employment checks. Within weeks HMRC stopped new enquiries, reinstated an employment check and apologized; for the second rollout in March 2026 it gave claimants a month to prove eligibility before any suspension.

An AI case, alleged. In a proposed class action, the plaintiffs allege that UnitedHealth and naviHealth used an AI program, nH Predict, to deny coverage. The court’s September 18, 2026 order set February 16, 2027 as the deadline for the plaintiffs’ class-certification motion. These are allegations, not findings.

How each could have been prevented

The inquiries, courts and auditors that examined these cases name the same missing safeguards.

Are you at risk?

Ask these of any system whose output is treated as a finding about a person.

1. Is the output treated as proof on its own, or does a person check it against independent evidence before acting?
2. Does the person affected have to disprove the output, or does the organization have to show it is right?
3. Are known errors in the system disclosed to the people affected and to whoever decides their case?
4. Can the person see the rules, the data and the calculation that produced the decision?
5. Is there a route to a human who can change the decision, before or soon after it takes effect?
6. Are complaints, appeals and reversals counted and treated as evidence about the system, not only about the individual case?
7. Is the system tested regularly, and do changes to its data or rules go back through oversight before use?
8. Is someone independent of the operator watching the outcomes?

A “no” to the first two questions is the pattern the inquiries and courts found in the cases above.