A Check Counts Only If It Can Fail and Stop the Work

The stop path is part of the check

Maryland’s Inspector General for Education investigated and reported in November 2025. It found that the detection software and the monitoring protocols had operated within their designed parameters, and attributed the police response to human error and training gaps. Omnilert said its system “operated as designed”; the superintendent said the program “did what it was supposed to do”. By the investigation’s account, both were right about the detector.

The failure was in what happened after the check. The Inspector General found that:

• 15 people received the alert, five in the school system and ten in county police, and other recipients could bypass the delegated-authority protocol by phoning or texting police.
• Enlarging the image or watching the video on a phone could cause a recipient to miss the cancellation text.
• The school system’s delegation of authority did not address an alert that had already been canceled being passed to police as active.

Eighteen days later the sequence repeated at Parkville High School. Staff sent a cancellation, others in the notification chain continued the threat protocol “out of an abundance of caution”, and students were moved while police searched a building with no threat in it. Once police were involved, the Inspector General found, the school system lost control of the alert and the response.

Its recommendations were about the stop path, not the detector: inform police of a confirmed threat, with a default notification if the primary reviewers cannot respond promptly; have reviewers use the same app view instead of text messages; and hold what the report calls bi-annual training for school executive staff and police on the protocol. The school system said it would adopt them and had retrained administrators.

A check that cannot stop the next step is a notification, not a control.

A pass on the average case says little about the case that matters

The second way a check fails is quieter: it is tested on the work that is easy to get right.

In an invented example, an AI handles 980 routine requests correctly and gets all 20 requests that needed a specialist hold wrong. Its overall score is 98%. Its score on the cases that matter is zero.

NIST’s February 2026 report on statistical models for evaluating language models makes the point for benchmarks: evaluating solely in aggregate can give an incomplete understanding of a model’s performance, and a single score misses the differences between subdivisions such as task, category or language. NIST’s January 2026 draft on benchmark practice lists, as an emerging practice, deciding whether to measure average-, best- or worst-case behavior, noting that high-stakes risk assessments may focus on best- or worst-case behavior while comparison shopping may focus on the average. In banking, the 2026 interagency model risk guidance describes outcomes analysis against an organization’s performance thresholds and ongoing monitoring as conditions change; it covers traditional models and non-generative, non-agentic AI, and puts generative and agentic AI outside its scope. US federal agencies must test high-impact AI before deployment in ways that reflect expected real-world outcomes, including by querying a vendor’s system where they cannot see its code.

A second AI tends to repeat the first one’s mistakes

Asking a second model to check the first feels independent. The 2025 research suggests it often is not.

A study of more than 350 language models, presented at ICML 2025, found that when two models were both wrong, they chose the same wrong answer about 60% of the time on one benchmark, against one in three by chance. Models from the same provider or architecture shared more errors, and more accurate models shared more errors even after allowing for that. When a model was used as a judge, it inflated the accuracy of less accurate models, marking wrong answers correct when both made the same mistake. A second 2025 study found that a judge model’s scores tracked how similar the judged model was to it, and that models’ mistakes are becoming more alike as they improve.

Agreement can also repeat a wrong definition. In an invented example, an AI reports a 90% success rate: 90 tasks completed out of 100 started. A second AI confirms the division. The agreed measure was tasks completed on time out of tasks due, 90 out of 120, or 75%. Both checks did the arithmetic. Neither checked the definition. A useful second check uses a different route to the answer: the source, the definition, a known-answer case or the final artifact.

How to build a check that can fail

1. Name the failures that must stop the work, before testing. A wrong answer on a specialist case is a different failure from a slow answer on a routine one.
2. Score the critical cases separately and report them beside the overall result.
3. Plant failures. Put known errors into test cases and confirm the check catches them for the right reason.
4. Prove the check can fail. In a test environment, weaken the check on purpose; the test should then fail. If it still passes, it was not testing what you thought.
5. Give the check a different route from the work it checks, and do not count a second model’s agreement as verification.
6. Make the stop path work end to end. One channel, one view for every reviewer, one person or rule that can stop the output, and no way around it by phone or text.
7. Retest when anything changes: the model or version, the sources, the prompts, the permissions, the connected tools or the population.

Where this applies across the centers

• Finance: a reconciliation or anomaly tool whose exceptions are reviewed, but whose cleared items flow straight into the ledger.
• People teams: a screening tool tested on overall accuracy but not on candidates who need accommodations or fall outside the usual profile.
• Buying AI: a vendor’s accuracy figure, which is an average; ask for results on the cases you care about and how the product’s alerts can be canceled.

Are you at risk?

• When a reviewer rejects or cancels an AI output, does anything stop the people or systems downstream from acting on it?
• Can someone in the chain bypass the agreed escalation by phoning or messaging directly?
• Is the AI use’s test result reported only as one overall figure?
• Is a second AI’s agreement treated as a check?
• Has the check ever been shown to fail when it should?
• Was the use retested after the last model, source or permission change?

Show that the error can stop the work

Trust a check because you have seen it catch the error that matters and stop what comes next. Test the critical cases on their own, plant the failure, prove the check can fail, and make sure a “stop” reaches everyone who could act.