Eminence Growth Solutions, home
Menu

AI failures · Misrepresented AI capability

Pieces Technologies: Texas Attorney General Settlement Over Hallucination-Rate Accuracy Claims for Hospital Generative AI (2024)

Verified Checked against its sources on 3 sources quoted

Where it happened
Texas, United States
Organization
Pieces Technologies, Inc.
Date
Assurance of Voluntary Compliance signed August 5, 2024 (Pieces) and August 21, 2024 (State); petition for court approval filed August 21, 2024 in Dallas County district court (Cause No. DC-24-13476); announced September 18, 2024; a law-firm paper cites an Agreed Final Order dated October 2, 2024
Function
Product marketing and advertising
Work stage
Advertising the accuracy of clinical generative AI products
System
Pieces generative AI products for summarizing patient records and drafting clinical notes in hospitals
Technology
Generative AI
Type of failure
Misrepresented AI capability
Caught by
A regulator

Summary

On September 18, 2024, the Texas Attorney General announced a settlement with Pieces Technologies, Inc., a Dallas-area company whose generative AI products summarize patients' condition and treatment and draft clinical notes for hospital staff; according to the release, at least four major Texas hospitals were sending it patient data in real time. Pieces had advertised its products' accuracy with metrics including a "critical hallucination rate" and "severe hallucination rate" of less than .001% and less than 1 per 100,000. The State alleged that these representations may have violated the Texas Deceptive Trade Practices Act because they were false, misleading or deceptive; the Attorney General's release said its investigation found the metrics were likely inaccurate. Pieces denied wrongdoing and maintained that it had accurately represented its hallucination rate. Under the Assurance of Voluntary Compliance, filed for approval in Dallas County district court on August 21, 2024 (a later law-firm paper cites an Agreed Final Order dated October 2, 2024), Pieces must define any accuracy metric it advertises and explain how it is calculated, or have its claims substantiated by an independent auditor; must not misrepresent its products' accuracy, testing or training data; and must give customers documentation of known limitations, risks and misuses. The assurance runs five years from court approval, provides for no payment, and resolves allegations without any finding of violation or of patient harm.

Consequence

  • Assurance of Voluntary Compliance filed for court approval on August 21, 2024; it runs five years from the court's approval
  • Pieces must disclose the meaning and calculation of any accuracy metric it advertises, or have claims substantiated by an independent auditor
  • Pieces must give customers documentation of known limitations, risks and misuses of its products

Who caught it A regulatorFound outside the organization

Analysis

What could have been done

Based on our analysis of publicly available information and reporting of the event, the following represent potential safeguards that may not have been present or fully in effect:

Evidence behind claims about AI

Public statements about AI capability are backed by evidence before they are made.

Pieces advertised critical and severe hallucination rates of less than .001% and less than 1 per 100,000, which the State alleged may have been false or misleading and which the assurance now requires to be defined and explained, or audited, whenever advertised.

What would likely have prevented it

  • Publish any accuracy or error-rate metric for an AI product together with its definition and the method used to calculate it, or have it independently audited.
  • Give customers written documentation of the product's known limits and misuses alongside any accuracy claim, so the claim is read in context.

What would likely have caught it earlier

  • Have someone outside the product team reconcile each advertised accuracy figure to the underlying test data and method before release.

Key dates

  1. September 18, 2024Event
  2. October 30, 2026Checked against sources
  3. November 3, 2026Checked against sources

Developments are added when the record is reviewed. Next review: January 30, 2027.

Evidence

Each passage is quoted from its source. The record shows the date it was last checked.

  1. Texas Attorney General Ken Paxton secured a first-of-its-kind settlement with a Dallas-based artificial intelligence healthcare technology company called Pieces Technologies.

    Supports: settlement announced

    The agreement resolves allegations that the company deployed its products at several Texas hospitals after making a series of false and misleading statements about the accuracy and safety of its products.

    Supports: the agreement resolves allegations

    At least four major Texas hospitals have been providing their patients’ healthcare data in real time to Pieces so that its generative AI product can “summarize” patients’ condition and treatment for hospital staff.

    Supports: hospital deployment

    September 18, 2024 | Press Release

    Supports: announcement date

    Attorney General Ken Paxton Reaches Settlement in First-of-its-Kind Healthcare Generative AI Investigation Office of the Attorney General of Texas (press release, September 18, 2024) Primary source

  2. Respondent advertised and marketed the accuracy of its products and services by claiming that they have a “critical hallucination rate” and “severe hallucination rate” of “<.001%” and “<1 per 100,000.”

    Supports: the hallucination-rate claims

    The State alleges that Respondent’s representations regarding its generative AI products may have violated the DTPA because they were false, misleading, or deceptive.

    Supports: the State's allegation

    Respondent denies any wrongdoing or liability and contends that it has not engaged in conduct that violates Texas law, including the DTPA, and that it has accurately set forth and represented its hallucination rate.

    Supports: Pieces denies the allegations

    Respondent must clearly and conspicuously disclose (1) the meaning or definition of such metric, benchmark, or similar measurement;

    Supports: assurance term: disclose the meaning of accuracy metrics

    alternatively, Respondent may retain an independent, third-party auditor

    Supports: assurance term: third-party auditor alternative

    Respondent must provide all its current and future customers, in connection with any of its products or services, documentation that clearly and conspicuously discloses any known or

    Supports: assurance term: customer documentation of risks and limits

    This Assurance shall automatically terminate five (5) years after its effective date.

    Supports: five-year term

    The effective date of this Assurance is the date the Assurance is approved by the Court.

    Supports: effective on court approval

    Dated: August 21, 2024

    Supports: petition dated and filed August 21, 2024

    In the Matter of State of Texas and Pieces Technologies, Inc.: Petition for Approval and Entry of Assurance of Voluntary Compliance, with the Assurance (Dallas County District Court, Cause No. DC-24-13476) Office of the Attorney General of Texas (file-stamped court filing) Primary source

  3. See Agreed Final Order, In the Matter of the State of Texas and Pieces Technologies, Inc., Cause No. DC-24-13476 (191st Judicial District, Dallas County Oct. 2, 2024).

    Supports: A legal paper cites an Agreed Final Order in this cause dated October 2, 2024

    Taking Stock of the Texas AG's New Focus on Privacy, Technology, and AI Association of Corporate Counsel (hosted paper) Reporting or analysis

Suggested citation. Eminence Scan, “Pieces Technologies: Texas Attorney General Settlement Over Hallucination-Rate Accuracy Claims for Hospital Generative AI (2024)”, Eminence Growth Solutions, last verified November 3, 2026, https://eminencegrowthsolutions.com/tools/eminence-scan/record/fail-110/.

Using our research: you may read and quote Eminence Scan with credit to “Eminence Scan, Eminence Growth Solutions” and a link; all other rights reserved. Terms: Using our research.

Spotted an error, or a development we have missed? Report a correction. How we check records: research methodology.

Stay Connected

Follow what matters
to you and your work.

AI Cost & Governance | AI Talent & People | Vendor Intelligence
Eminence Scan | EmTrace | Eminence Edge

Choose updates by center and tool.

Choose your alerts

Research and practical tools, chosen by you.

Sign in with your email address. There is no password.