Eminence Growth Solutions, home
Menu

Eminence Growth Solutions

Practice note

What an AI Pilot Has to Prove

In METR’s 2025 trial, experienced developers expected AI tools to save them 24% of their time and were measured taking 19% longer. A pilot has to measure the completed process, test the review and the authority around the tool, and end with a clear decision to expand, keep testing or stop.

Download PDF
Watercolor of several measured streams converging on one outcome point, with a small orange gap.

Key takeaways

  • Write the expansion decision and acceptance criteria before examining the pilot’s results.
  • Compare completed work, review, corrections and operating cost against an equivalent baseline.
  • Test the review, authority limits and fallback, then decide whether to expand, keep testing or stop.

In early 2025 METR, an AI research organization, ran a randomized trial with 16 experienced open-source developers working on 246 real tasks in mature projects they knew well. Before starting, the developers expected AI tools to cut their time by 24%; afterwards they believed the tools had cut it by 20%. Measured, they took 19% longer when AI was allowed. On the recorded subset of tasks, their time moved from writing code to prompting, waiting for and reviewing AI output, and they accepted fewer than 44% of the AI’s generations.

METR has since said those results are out of date and that developers are likely more sped up by AI in early 2026. What has not changed is the gap between feeling faster and being faster: in its 2026 survey METR repeated that people in the 2025 study overestimated AI’s effect on their time by 40 percentage points, and its 2025 paper argues for field experiments with robust outcome measures rather than reliance on expert forecasts or developer surveys.

The study measured experienced software developers working in familiar open-source projects. It does not estimate the productivity effect for finance teams, other jobs or current tools. Its practical contribution here is the measurement question: does the completed task improve?

A pilot that times only the draft can report a saving while the whole task gets slower. Define success as an improvement in the completed finance process, with evidence strong enough to support reliance. A faster draft is worth measuring; so are the review, rework and running costs it takes to make the draft usable.

Start with a decision the pilot can resolve

Choose one workflow, one accountable owner and one decision about expansion. Should an assistant become the regular first-draft preparer of commentary for a defined set of accounts? Should a matching feature be used within an approved tolerance? Should an agent receive a particular write permission?

Those are different decisions. A tool that drafts commentary well has not shown that it should post entries, and a clean reconciliation on one entity says little about the consolidation.

Write down the acceptance criteria before seeing the results: the outcome sought, the defects that would matter, the evidence to retain, and the conditions that would stop or limit the pilot. The pilot then follows the later phases of the close playbook’s rollout sequence: validate in parallel, set release conditions, then monitor and expand selectively.

Measure the whole task against a credible baseline

The gain and the cost often sit in different steps.

Record how the current process works, including its defects. Compare the AI-supported process on a closely similar scope: the same reporting purpose, entities, source populations and acceptance criteria. Note any difference that prevents a fair comparison.

Follow the work through to an approved result; do not stop the clock when the first output appears. Count effort that moves to other people, such as a manager who now checks the draft, and check volume and adoption before annualizing a small convenience.

A pilot that compares the whole task covers six dimensions:

Exhibit · Pilot evidence

Six lenses. One comparable task. One expansion decision.

Judge the complete workflow against the same purpose, scope and acceptance criteria. A gain in one dimension does not settle the other five.

Six evidence lenses around one comparable workflowThe same purpose, scope and acceptance criteria connect completed output, total effort, elapsed time, defects and coverage, evidence and operating cost. This is a qualitative evidence framework, not a score. Completed output: Whether it answers the required question and meets the same acceptance criteria. Total effort: Preparation, checking, investigation, correction, approval and continuing maintenance. Elapsed time: Whether the workflow reaches an approved result sooner, including queues and escalations. Defects and coverage: Wrong results, missing items, unresolved exceptions and cases the system could not process. Evidence: Whether the source, method, checks and approval can be retrieved. Operating cost: Incremental charges and commitments, alongside the internal effort required.01Completed outputFit for purpose?02Total effortAll effort counted?03Elapsed timeApproved sooner?04Defects and coverageWhat failed or is missing?05EvidenceCan we retrieve it?06Operating costFull cost supported?ONE WORKFLOWSame purposeSame scopeSame criteria

Use the evidence: expand within a defined scope, continue a bounded test or stop.

  1. 01 · Completed outputFit for purpose?
  2. 02 · Total effortAll effort counted?
  3. 03 · Elapsed timeApproved sooner?
  4. 04 · Defects and coverageWhat failed or is missing?
  5. 05 · EvidenceCan we retrieve it?
  6. 06 · Operating costFull cost supported?
Read the six measurement definitions
  1. 01

    Completed output

    Whether it answers the required question and meets the same acceptance criteria.

  2. 02

    Total effort

    Preparation, checking, investigation, correction, approval and continuing maintenance.

  3. 03

    Elapsed time

    Whether the workflow reaches an approved result sooner, including queues and escalations.

  4. 04

    Defects and coverage

    Wrong results, missing items, unresolved exceptions and cases the system could not process.

  5. 05

    Evidence

    Whether the source, method, checks and approval can be retrieved.

  6. 06

    Operating cost

    Incremental charges and commitments, alongside the internal effort required.

CoE analysis of the dimensions already defined in this article. The lines connect the evidence to one workflow; they do not prescribe a measurement sequence or assign numerical scores.

Count capacity, cash and better decisions separately

Keep three benefit claims apart.

  • Capacity released: Time becomes available for other work. The two hours in the illustration are capacity released.
  • Cash savings: These require an actual change in spending. The time released becomes a cash saving only if spending changes.
  • Better decisions or controls: These need their own evidence.

Count each improvement once. Do not turn an estimate of how exposed a role is to AI into a headcount target.

The strongest business case may show a smaller percentage, with a much more credible account of where the value went.

Use The AI Spend Stack to price the pilot layer by layer. The value side comes from the pilot’s own measurements.

Test the hard cases and the review around them

A pilot made of clean, routine transactions tells you little about an exception-heavy close. Include the difficult cases that matter in this workflow: incomplete inputs, changed billing patterns, unusual account mappings, unsupported explanations, or a request outside the agent’s authority. Use controlled test data and an environment suited to the risk.

Run two tests, not one: whether the system produces an acceptable result, and whether the review around it catches an unacceptable one. For the second, use the planted-error exercise in Review Is a Control Only When It Can Catch the Error. A planted error that passes review is a weakness in the review, however well the tool handles ordinary cases.

The COSO-commissioned publication Achieving Effective Internal Control Over Generative AI calls for human corroboration proportionate to the risk and treats generative AI outputs as assertions requiring evidence. The test set here applies that principle to a pilot.

Test the authority and the fallback

If the feature can change a record or start a transaction, test the limit in the system that executes the action. Confirm that restricted actions are rejected, that approvals cannot be bypassed, and that the authoritative records show what happened.

OWASP recommends limiting the permissions an AI model’s extensions hold in other systems to the minimum necessary, and enforcing authorization in those downstream systems rather than relying on the model to decide whether an action is allowed.

Establish how the process continues when the tool is unavailable, an input is incomplete, or usage reaches a commercial limit. Test the fallback before a reporting deadline depends on it.

Expand only the reliance the pilot supports

End with one of three decisions: expand within a defined scope, continue a bounded test to resolve a named uncertainty, or stop. Record the basis for the decision; finishing the pilot is not the same as approving the tool.

A useful expansion decision names the authorized tasks, data, permissions, required review, owner and monitoring. The COSO-commissioned publication treats prompts, system prompts, retrieval connectors and transformation rules as governed configurations with version history, approval workflows and rollback plans. A change to any of them is a reason to reassess what the pilot showed.

Before the next stage, answer one question in plain language: what has this pilot demonstrated that we can now rely on, and under what conditions?

Sources and further reading

Review status

Orientation, not accounting or audit advice; apply it with your auditors and advisers.

Published by Eminence Growth Solutions
https://eminencegrowthsolutions.com/coe/cost-governance/what-an-ai-pilot-has-to-prove/

Help improve this page

Better intelligence comes from better information. You don’t need to have a fully researched recommendation or a formal correction. Sometimes a question, observation, or firsthand experience is enough to identify something worth investigating.

Questions, corrections and responses from organizations we cover: egsconnect@eminencegrowthsolutions.com

Stay Connected

Follow what matters
to you and your work.

AI Cost & Governance | AI Talent & People | Vendor Intelligence
Eminence Scan | EmTrace | Eminence Edge

Choose updates by center and tool.

Choose your alerts

Research and practical tools, chosen by you.

Sign in with your email address. There is no password.