AI failures · Misrepresented AI capability
Pieces Technologies: Texas Attorney General Settlement Over Hallucination-Rate Accuracy Claims for Hospital Generative AI (2024)
Verified Checked against its sources on 3 sources quoted
- Where it happened
- Texas, United States
- Organization
- Pieces Technologies, Inc.
- Date
- Assurance of Voluntary Compliance signed August 5, 2024 (Pieces) and August 21, 2024 (State); petition for court approval filed August 21, 2024 in Dallas County district court (Cause No. DC-24-13476); announced September 18, 2024; a law-firm paper cites an Agreed Final Order dated October 2, 2024
- Function
- Product marketing and advertising
- Work stage
- Advertising the accuracy of clinical generative AI products
- System
- Pieces generative AI products for summarizing patient records and drafting clinical notes in hospitals
- Technology
- Generative AI
- Type of failure
- Misrepresented AI capability
- Caught by
- A regulator
Summary
On September 18, 2024, the Texas Attorney General announced a settlement with Pieces Technologies, Inc., a Dallas-area company whose generative AI products summarize patients' condition and treatment and draft clinical notes for hospital staff; according to the release, at least four major Texas hospitals were sending it patient data in real time. Pieces had advertised its products' accuracy with metrics including a "critical hallucination rate" and "severe hallucination rate" of less than .001% and less than 1 per 100,000. The State alleged that these representations may have violated the Texas Deceptive Trade Practices Act because they were false, misleading or deceptive; the Attorney General's release said its investigation found the metrics were likely inaccurate. Pieces denied wrongdoing and maintained that it had accurately represented its hallucination rate. Under the Assurance of Voluntary Compliance, filed for approval in Dallas County district court on August 21, 2024 (a later law-firm paper cites an Agreed Final Order dated October 2, 2024), Pieces must define any accuracy metric it advertises and explain how it is calculated, or have its claims substantiated by an independent auditor; must not misrepresent its products' accuracy, testing or training data; and must give customers documentation of known limitations, risks and misuses. The assurance runs five years from court approval, provides for no payment, and resolves allegations without any finding of violation or of patient harm.
Consequence
- Assurance of Voluntary Compliance filed for court approval on August 21, 2024; it runs five years from the court's approval
- Pieces must disclose the meaning and calculation of any accuracy metric it advertises, or have claims substantiated by an independent auditor
- Pieces must give customers documentation of known limitations, risks and misuses of its products
Who caught it A regulatorFound outside the organization
Analysis
What could have been done
Based on our analysis of publicly available information and reporting of the event, the following represent potential safeguards that may not have been present or fully in effect:
Evidence behind claims about AI
Public statements about AI capability are backed by evidence before they are made.
Pieces advertised critical and severe hallucination rates of less than .001% and less than 1 per 100,000, which the State alleged may have been false or misleading and which the assurance now requires to be defined and explained, or audited, whenever advertised.
What would likely have prevented it
- Publish any accuracy or error-rate metric for an AI product together with its definition and the method used to calculate it, or have it independently audited.
- Give customers written documentation of the product's known limits and misuses alongside any accuracy claim, so the claim is read in context.
What would likely have caught it earlier
- Have someone outside the product team reconcile each advertised accuracy figure to the underlying test data and method before release.
Key dates
- September 18, 2024Event
- October 30, 2026Checked against sources
- November 3, 2026Checked against sources
Developments are added when the record is reviewed. Next review: January 30, 2027.
Evidence
Each passage is quoted from its source. The record shows the date it was last checked.
-
Texas Attorney General Ken Paxton secured a first-of-its-kind settlement with a Dallas-based artificial intelligence healthcare technology company called Pieces Technologies.
Supports: settlement announced
The agreement resolves allegations that the company deployed its products at several Texas hospitals after making a series of false and misleading statements about the accuracy and safety of its products.
Supports: the agreement resolves allegations
At least four major Texas hospitals have been providing their patients’ healthcare data in real time to Pieces so that its generative AI product can “summarize” patients’ condition and treatment for hospital staff.
Supports: hospital deployment
September 18, 2024 | Press Release
Supports: announcement date
-
Respondent advertised and marketed the accuracy of its products and services by claiming that they have a “critical hallucination rate” and “severe hallucination rate” of “<.001%” and “<1 per 100,000.”
Supports: the hallucination-rate claims
The State alleges that Respondent’s representations regarding its generative AI products may have violated the DTPA because they were false, misleading, or deceptive.
Supports: the State's allegation
Respondent denies any wrongdoing or liability and contends that it has not engaged in conduct that violates Texas law, including the DTPA, and that it has accurately set forth and represented its hallucination rate.
Supports: Pieces denies the allegations
Respondent must clearly and conspicuously disclose (1) the meaning or definition of such metric, benchmark, or similar measurement;
Supports: assurance term: disclose the meaning of accuracy metrics
alternatively, Respondent may retain an independent, third-party auditor
Supports: assurance term: third-party auditor alternative
Respondent must provide all its current and future customers, in connection with any of its products or services, documentation that clearly and conspicuously discloses any known or
Supports: assurance term: customer documentation of risks and limits
This Assurance shall automatically terminate five (5) years after its effective date.
Supports: five-year term
The effective date of this Assurance is the date the Assurance is approved by the Court.
Supports: effective on court approval
Dated: August 21, 2024
Supports: petition dated and filed August 21, 2024
-
See Agreed Final Order, In the Matter of the State of Texas and Pieces Technologies, Inc., Cause No. DC-24-13476 (191st Judicial District, Dallas County Oct. 2, 2024).
Supports: A legal paper cites an Agreed Final Order in this cause dated October 2, 2024
Suggested citation. Eminence Scan, “Pieces Technologies: Texas Attorney General Settlement Over Hallucination-Rate Accuracy Claims for Hospital Generative AI (2024)”, Eminence Growth Solutions, last verified November 3, 2026, https://eminencegrowthsolutions.com/tools/eminence-scan/record/fail-110/.
Using our research: you may read and quote Eminence Scan with credit to “Eminence Scan, Eminence Growth Solutions” and a link; all other rights reserved. Terms: Using our research.
Spotted an error, or a development we have missed? Report a correction. How we check records: research methodology.
