Interviewing for AI Judgment: Ask Candidates to Catch the Error

The decision this note informs

What to add to the interview process for roles that will produce, use or review AI-assisted work, such as analysts, reviewers, managers and anyone who signs off, and how to score it consistently.

Asking about AI is not testing it

Questions about which tools a candidate has used, or how they write prompts, measure familiarity. They do not measure the capability that matters when AI drafts the work: recognizing when a polished answer is wrong, and knowing which part to check.

Self-assessment is a poor guide here. In an experiment in which 246 people used ChatGPT on logical-reasoning problems, they scored three points above a norm population but overestimated their performance by four, and a second study of 452 people replicated the result. People with more technical knowledge of AI were more confident but less accurate in judging their own performance. In a study of 83 university students, an objective test of generative AI literacy predicted performance on an AI-assisted task, while a self-reported literacy scale did not.

The underlying problem is well documented. In an experiment with 758 early-career Boston Consulting Group consultants, co-authored by BCG researchers, AI-assisted recommendations on a task outside the AI’s capabilities were more often wrong, yet graders rated them more coherent and persuasive. In the same study, consultants also given a supplementary prompt-engineering overview did worse on that task than those with AI alone, a difference significant only at the 10% level. For the wider argument, see The Competence Inversion.

Employers are testing different approaches

• McKinsey has added the use of AI to its interviews for junior consultants since the end of 2025, according to the Financial Times. “We assess judgment and how they use it. We don’t test how to prompt or how many tools they know,” a partner told the FT. Candidates bring their own AI tools, and the test is one input among several.
• Envoy: a candidate for a senior accounting role was asked to discuss the prompts and data used to create technical memos, the FT reported in an article quoting Envoy’s chief financial officer.
• Canva says its AI-assisted coding interview, piloted for engineering roles, evaluates whether candidates can identify and fix issues in AI-generated code, and that it tells candidates in advance that they will be expected to use AI. It reports that candidates with little AI experience often lacked the judgment to tell when AI’s suggestions were poor.
• Amazon says it has reworked its software-engineer assessments so that candidates work with an AI assistant.
• Goldman Sachs takes the opposite approach for campus applicants: a spokesperson told Fortune it wants “to hear from our applicants in their own voice”, and its recruitment team told students that external sources, including ChatGPT, are prohibited during interviews.

At least one assessment vendor is moving the same way. Codility, which sells technical assessments, says its “AI for Business” simulations score how people catch AI hallucinations and errors; one example asks the candidate to verify an AI budget report containing three errors. It also lets interviewers tell its AI coding agent to slip in a bug, to see whether candidates review the code. The pages read publish no evidence that these simulations predict job performance.

A four-stage exercise

Test review directly, with a sample of the work. Give candidates a short task from the role that requires AI, and score four stages of how they handle it. The stages below are CoE analysis, mapped to two published frameworks: the AI Fluency Framework, developed by two professors and published with Anthropic, and the US Department of Labor’s AI Literacy Framework.

Most of the score should sit in stage 3. To make it observable, give every candidate the same AI-produced draft containing a small number of planted errors alongside correct content.

Give every candidate the same AI. Use one approved AI tool for every candidate, or give every candidate identical AI output to review. Equal access keeps the scoring consistent and keeps the employer’s case material out of personal accounts. McKinsey lets candidates bring their own tools; in a scored exercise, candidates’ own tools fit practice rounds only.

Illustrative example (fictional), for a financial analyst role: a two-page AI-drafted variance commentary. It has three planted errors: a cited source that does not exist, a percentage that does not follow from the figures in the table, and a causal explanation the data do not support. Several other statements are correct and should be accepted. The candidate has 45 minutes and the approved AI tool, and submits their plan, the prompts they used, their marked-up review and a short note to the decision-maker.

The same design can be adapted for other roles:

• HR: an AI-drafted job description or policy summary with a misstated legal requirement.
• Consulting or strategy: an AI-produced market sizing built on an unsupported assumption.

Anchored scores (1 to 4), per stage:

Credit correct acceptance as well as rejection. A candidate who flags everything has not shown judgment, and the exercise should not reward suspicion for its own sake.

Score consistently. Use the same task, anchors and rules for every candidate for the role. A 1995 meta-analysis found that interrater reliability depended on interviewer training and on standardized questions, standardized evaluation and how multiple ratings are combined, with ratings useful when combined mechanically but not when combined subjectively. A 2013 meta-analysis found higher interrater reliability for panel interviews than for separate interviews (.74 against .44). A 2024 meta-analysis found interview ratings of task and contextual performance had similar validity, and that structured scoring might particularly help for contextual performance.

Run it as a pilot first. No employer or vendor cited here publishes evidence that a planted-error exercise predicts later job performance, or evidence on its adverse impact. The number of errors, the time limit, the format and the effect of AI assistance all need testing in your own setting. Tell candidates in advance what the exercise involves and which AI tool they will use. Track selection rates by group from the start: the federal Uniform Guidelines treat a group’s selection rate below four-fifths of the highest group’s as generally regarded as evidence of adverse impact.

Where the exercise fits among selection methods

A 2022 re-analysis of selection research revised earlier validity estimates downward. Structured interviews ranked first, at .42; work-sample tests fell from .54 to .33; and unstructured interviews fell from .38 to .19. The authors note that the strongest predictors were job-specific measures. Those estimates are debated: other researchers argue that the re-analysis underestimates validity, and the original authors reply that the data needed for the correction are usually unavailable in meta-analysis and that the effect is usually small.

A review exercise is a work sample and belongs inside a structured process, not in place of one. It adds a direct look at one capability the rest of the process does not test.

We found no study validating this exercise as a hiring tool. The closest published study we found, a 2026 validation of an assessment of performance under AI advice that includes detecting AI errors, used 780 employed adults in Türkiye rather than job applicants, measured its outcome in the same session, and its authors say the tool should not currently be used as a stand-alone basis for hiring decisions.

The legal frame (orientation for U.S. employers)

A review exercise is a selection procedure. The federal Uniform Guidelines define a selection procedure to include performance tests and informal interviews used as a basis for an employment decision.

Disparate impact. Under Title VII, a practice shown to cause a disparate impact on a protected basis is unlawful if the employer cannot show it is job-related for the position and consistent with business necessity, or if the employer refuses an alternative practice the complaining party has shown. The 2024 edition of the United States Code shows no amendment to that text since 1991. The Uniform Guidelines treat a selection rate for any race, sex or ethnic group below four-fifths of the highest group’s rate as generally regarded as evidence of adverse impact. They say that the closer a procedure is to a work sample, the stronger the basis for showing content validity, and that a test sampling a work product should approximate the work situation.

The federal posture is changing.

• An executive order of April 2025 makes it U.S. policy to eliminate disparate-impact liability to the maximum degree possible and directs agencies to deprioritize its enforcement, naming Title VII.
• In June 2026 the Justice Department’s Office of Legal Counsel concluded that the Uniform Guidelines rest on an unconstitutional reading of Title VII, and described aptitude tests as presumptively job-related.
• The EEOC’s 2026 regulatory agenda lists a final rule rescinding the Guidelines’ interpretive portions, with final action targeted for November 2026.
• OPM removed references to the Guidelines from federal personnel rules from July 31, 2026, and a Labor Department final rule published in August 2026 removes its copy for federal contractors, effective October 26, 2026.
• The Guidelines remain in the Code of Federal Regulations; the eCFR text is current to October 15, 2026.

These changes come from the executive branch. They do not amend the statute. How courts will treat them in private suits is a question for counsel. State and local rules on automated hiring tools, and EU rules, are tracked in AI in Hiring: Standards and Regulatory Watch.

Disability accommodation. The ADA treats as discrimination a test that screens out people with disabilities unless it is job-related and consistent with business necessity, and a failure to administer tests so that results reflect the skill measured rather than an impairment. EEOC’s codified guidance lists more time to complete a test among possible accommodations, and says the obligation does not apply where the impaired skill, such as speed, is what the test is designed to measure. The guidance turns on whether speed is a skill the test measures, so a timed exercise needs to say which.

EEOC removed its 2022 and 2023 technical assistance on AI in employment assessments from its website in late January 2025.

Candidate AI use and identity

Decide in advance which AI candidates may use during the exercise, the approved tool or none, and say so. In two studies of applicants taking a standardized pre-hire assessment, self-reported generative AI use rose between 2024 and 2025, and warnings about consequences reduced reported use.

Remote interviews also carry identity risk. Two vendor surveys published in 2025 give a sense of scale:

• Greenhouse, which sells hiring software, reported that 65% of the US hiring managers it surveyed had caught applicants using AI deceptively, including 18% who had seen deepfakes, and that 39% were holding more in-person interviews.
• Checkr, which sells background checks, reported that 35% of the 3,000 American managers involved in hiring it surveyed said someone other than the listed applicant had taken part in a virtual interview.

The FBI has warned since 2022 of deepfakes and stolen identities used to apply for remote roles. In October 2026 it reported that North Korean IT workers pay people to appear on camera and take part in interviews on their behalf. It also warns that aggressive controls carry high risks of false positives that harm legitimate applicants. A proportionate identity check or supervised stage may help confirm who completes the exercise. Offer an accommodation route, and resolve doubts without treating a suspicion as proof.

Questions leaders should ask

1. For which roles does reviewing AI-produced work affect decisions others rely on?
2. Do we test that capability directly, or only ask about tools and certificates?
3. Is the exercise built from the actual work of the role, and could we explain why each planted error matters to the job?
4. Are the anchors fixed in advance and applied by trained raters, ideally more than one?
5. Do we monitor selection rates by group, and have employment counsel reviewed the exercise?
6. Have we stated whether speed is measured, and how candidates request accommodations?
7. Have we told candidates which AI tool they will use, and how we confirm who is taking part?