Eminence Scan
AI failures
Documented cases where AI, or an automated system doing the same job, caused harm, a loss, a sanction or a retraction. Each case is checked against court records, regulator releases or established reporting. Each incident carries a lesson: the control that was missing, what would likely have prevented it, and what would have caught it sooner. Security and operational lapses at AI companies and in AI tools are tracked too, labeled as such and kept separate from incident counts.
What the cases show
17%
of verified failures name the organization’s own review or monitoring among those who found the error. In the rest, only outsiders are recorded: a court, a client, a journalist, a researcher or the people affected.
Fabricated contentOtherHarmful outputIncorrect automated decisionMisrepresented AI capabilityAgent acted beyond its limitsUnauthorized commitmentData exposureModel failureIdentity deception
What the cases teach
- Findings reportMost Documented AI Failures Were Found by Someone ElseOf the 60+ AI and automation failures in the Eminence Scan record documented well enough to verify, as of October 2026, only 16% name the organization's own people among those who found the problem, and in all but one of those the output had already been acted on. Journalists, courts and the people affected found most of the rest.
- InsightWhen a Report Cites Sources That Do Not ExistSince 2025, invented citations have surfaced in a national AI policy, a White House report, government-commissioned consulting reports, professional-firm research, and expert testimony, and invented quotations in orders drafted in judges' chambers. Many had passed through review. Almost every time, someone outside found the problem.
- InsightWhen the System Is Presumed RightSome of the most damaging automated-decision failures on record involve little or no AI. Horizon, Robodebt, Michigan's fraud system and, as recently as 2025, a UK benefit check all treated system output as proof and left people to disprove it. The cases show what was missing, and what would have caught the errors sooner.
- InsightWhen the Claim About AI Is the FailureRegulators are not policing AI as a technology. They are policing what companies say it does. In 2025 and 2026 the FTC finalized orders over an AI detector sold as 98% accurate, a "robot lawyer" it said was never tested against human lawyers, and an "AI-powered" ad service it said did not listen to anything, and the SEC has charged a company and a founder over AI that was really people.
Where
Records12–45–910 or moreNone
All records
Northflank: Customers Told to Assume Environment Variables Were Decrypted; Unauthorized Use of AI API Keys Reported
BleepingComputer reports that Northflank, a cloud platform for deploying and managing applications, disclosed a security incident on October 30, 2026 and warned customers to rotate environment variables and API keys after receiving a threa…
Security lapse
Cursor: Extension Marketplace Served a Modified Claude Code Extension That Attempted an SSH Backdoor
Cursor says that between about 2:27 am and 1:35 pm PT on October 29, 2026, a maliciously modified version of Anthropic's Claude Code extension, versions 2.1.312 and 2.1.313, was available in the Cursor VS Code Extension Marketplace.
Security lapse
Command Code: Company Links a Security Incident to a Distillation Attack and Says It Revoked All API Keys
Command Code says in a post on X, dated October 28, 2026, that it detected and stopped a security incident linked to a large-scale distillation attack targeting Anthropic and OpenAI models.
Security lapse
Warp: Attack on a Third-Party Vendor Exposed Agent Decision Logs and Telemetry
BleepingComputer reports that agent decision logs and telemetry data belonging to users of Warp, an agentic development environment, were exposed in an attack on a third-party vendor in Warp's infrastructure.
Security lapse
LiteLLM: Compromised Version 1.107.0 Published Through Its Release Pipeline (October 2026)
LiteLLM says its release pipeline published a compromised litellm 1.107.0 to PyPI and to its container registries on October 21, 2026.
Security lapse
Railway: Attack Exposed Customers' Encrypted Environment Variables, With a Subset Decrypted
Railway, a platform for deploying applications, says it became aware on October 20, 2026 of a sustained, targeted campaign to attack and breach its services, and believes the attack ran from October 20 at 13:58 UTC to October 22 at 03:32 U…
Security lapse
OpenCode: Breach of Hosted Console Exposed Linked AI Provider Keys; Maintainer Says 1.4% of Console Users Were Affected
BleepingComputer reports that OpenCode, an open-source AI coding project, notified some users that API keys they had stored for external providers were exposed in a security breach.
Security lapse
Lightning AI: Unauthorized Access to Credentials on the Lightning Platform
Lightning AI says that on October 15, 2026 at 11:00 PM UTC it identified a security incident on the Lightning Platform involving unauthorized access to credentials, and that it revoked all user access to the platform on October 16 at 1:00…
Security lapse
Zeabur: Leaked internal credentials used to reach its shared cluster and platform database
Zeabur, a hosting platform, says that on October 14 to 15, 2026 an attacker used leaked internal credentials to break into its shared cluster, some Dedicated Servers and internal systems, and eventually read its platform database, which ho…
Security lapse
Andromeda: Unauthorized Access to Credentials in Its Environment
Andromeda says that on October 13, 2026 it identified a security incident involving unauthorized access to credentials in its environment, disabled the compromised accounts that day, and has observed no further unauthorized activity since.
Security lapse
Fireworks AI: Unauthorized Use of Internal Credentials Reached Environment Variables in Customer Evaluation Jobs
Fireworks AI says it identified, on October 6, 2026, a security incident involving unauthorized use of its internal credentials, and that an unauthorized third party accessed environment variables used in evaluation jobs in a limited numbe…
Security lapse
NVIDIA Brev: Customer Notice Says a Security Incident May Have Exposed an SSH Key (October 2026)
Tom's Hardware reports that users described unauthorized activity on NVIDIA's Brev cloud development platform between October 4 and October 20.
Security lapse
UK AI Security Institute: Agents Under Test Took Unsanctioned Action Against Real People on GitHub, Including Fake Identities (2026)
The UK AI Security Institute (AISI) reports that on July 28, 2026 its security team detected unusual data transfers leaving its research systems during a routine cyber evaluation, which it had deliberately run with internet access and with…
Study
PwC Middle East: GPTZero Finds Fabricated Citations and Unsupported Framework Adopters in Four Thought Leadership Reports
A GPTZero investigation examined four PwC Middle East thought leadership reports published 2024-2026 and found four citations that met its definition of fabricated, including an invented academic paper on air quality in Riyadh, plus many c…
OpenAI: Models Under Evaluation Escaped a Test Environment and Breached Hugging Face's Production Systems (2026)
OpenAI says that in July 2026 models it was testing on a cyber-capability benchmark, with safeguards reduced for the evaluation, exploited a previously unknown vulnerability in a package-cache proxy to reach the internet, then chained vuln…
Anthropic: Shared Claude Conversations Became Searchable on Google; Search Engines De-indexed Them (2026)
Outlook Business reported in July 2026 that some publicly shared Claude conversations became searchable on Google.
Security lapse
Perplexity: Shared Perplexity Computer Files Reachable Through Search Engines
Protos reports that it could open dozens of full Perplexity Computer files belonging to assorted customers by searching Google, Bing and DuckDuckGo using the standard URL structure for shareable links, and that it reported the problem to P…
Security lapse
OpenAI: Experimental Model Gained Non-Public Access to an Australian Government Health Statistics Portal During Training (2026)
OpenAI says that in June 2026, during internal training and evaluation, an experimental internal-only model running without the full safeguards of its public products took actions it had not been authorised to take while researching govern…
Mistral AI: TanStack Supply-Chain Attack Produced Compromised SDK Releases
Mistral AI says it was affected by the Mini Shai-Hulud supply-chain attack, which was caused by a compromise of the third-party TanStack software package.
Security lapse
OpenAI: TanStack npm Supply Chain Attack Reached Two Employee Devices (2026)
OpenAI says two employee devices in its corporate environment were impacted by the TanStack npm supply chain attack.
Security lapse
Lightning AI: Compromised Versions of the lightning Package Were Available on PyPI for 42 Minutes
Lightning AI says that on April 30, 2026, an attacker captured PyPI credentials and used them to publish compromised versions 2.6.2 and 2.6.3 of its lightning package (PyTorch Lightning) to PyPI.
Security lapse
Anthropic: Report of Unauthorized Access to Claude Mythos Preview Through a Vendor Environment (2026)
TechCrunch reports that Bloomberg has reported that a private online forum, whose members have not been publicly identified, gained access to Claude Mythos Preview through a third-party vendor.
Security lapse
Generative AI Chatbots and Medical Misinformation — BMJ Open Audit Finds Half of Responses Problematic
A BMJ Open audit (Tiller et al., published April 14, 2026) put 50 health and medical questions to five consumer chatbots (Gemini, DeepSeek, Meta AI, ChatGPT and Grok) in February 2025, across cancer, vaccines, stem cells, nutrition and ath…
Study
AI-Enabled Fraud and Elder Fraud — FBI IC3 2025 Internet Crime Report
The FBI Internet Crime Complaint Center's 2025 annual report, its first with an AI section, records 22,364 complaints with an AI nexus and $893,346,472 in adjusted losses.
Statistic
Context.ai: Compromise of an AI Tool Used by a Vercel Employee Led to Access to a Vercel Environment
Vercel says its April 2026 security incident originated with a compromise of Context.ai, a third-party AI tool used by a Vercel employee.
Security lapse
OpenAI: Compromised Axios Library Ran in a macOS App-Signing Workflow (2026)
OpenAI says that on 31 March 2026 (UTC) Axios, a widely used third-party developer library, was compromised as part of a broader software supply chain attack, and that a GitHub Actions workflow in its macOS app-signing process downloaded a…
Security lapse
Anthropic: Claude Code Release Included Internal Source Code (2026)
Anthropic confirmed to Fortune and CNBC that some internal source code had been leaked within a Claude Code release.
Security lapse
Mercor: Security Incident Linked to the LiteLLM Compromise; Extortion Group Claims Data
TechCrunch reports that Mercor, an AI recruiting startup, confirmed a security incident linked to a supply-chain attack involving the open-source project LiteLLM.
Security lapse
Anthropic — Draft Announcement of Unreleased "Claude Mythos" Model Exposed Through Misconfigured Website CMS
Fortune reported on March 26, 2026 that a draft Anthropic blog post about an unreleased model, Claude Mythos, had been left in an unsecured, publicly searchable data store, among what Fortune said appeared to be close to 3,000 assets that…
Security lapse
LiteLLM — Supply Chain Attack Injects Credential-Stealing Malware into AI Gateway Library
On March 24, 2026, two malicious releases of LiteLLM, a widely used open-source Python gateway for calling LLM providers, were published to PyPI as versions 1.82.7 and 1.82.8.
Security lapse
Air AI: Court Order Bans Company and Owners from Marketing Business Opportunities; $18M Judgment, Mostly Suspended
On March 24, 2026, the FTC announced a settlement with Air AI, five related companies and their owners over its August 2025 complaint alleging they misled entrepreneurs and small businesses with false claims about business growth, earnings…
Internet Watch Foundation — Record 8,029 AI-Generated Child Sexual Abuse Items Found in 2025
The Internet Watch Foundation reported on March 24, 2026, that it identified 8,029 AI-generated images and videos of realistic child sexual abuse in 2025, a 14% increase on the previous year.
Statistic
Tesla Full Self-Driving — NHTSA Upgrades Reduced-Visibility Probe to Engineering Analysis EA26-002
NHTSA's Office of Defects Investigation opened Preliminary Evaluation PE24-031 on October 17, 2024, into Tesla Full Self-Driving crashes in reduced visibility, covering an estimated 2,410,002 vehicles and four crashes, one fatal.
Waymo Robotaxi Briefly Blocks Ambulance Responding to Austin Mass Shooting
After a mass shooting on West 6th Street in Austin, Texas, in the early hours of March 1, 2026, in which three people were killed and more than a dozen wounded, a Waymo robotaxi hailed for a pickup stopped across the road while executing a…
Google: Public Google API Keys Also Authenticated to Gemini, Truffle Security Reports (2026)
Truffle Security reported in February 2026 that Google API keys of the kind long described as safe to embed in public services such as Google Maps also authenticated to Gemini.
Security lapse
Fletcher v. Experian: Fifth Circuit Sanctions Counsel $2,500 for AI-Drafted Reply Brief With Fabricated Quotations (2026)
Robert Fletcher sued Experian and the lender Bridgecrest under the Fair Credit Reporting Act, alleging that an identity thief had opened an automobile finance account in his name.
Ars Technica Retracts Article Containing AI-Fabricated Quotations Attributed to Scott Shambaugh
On February 13, 2026, Ars Technica published an article about an AI agent that had written a hit piece on open-source maintainer Scott Shambaugh.
Lovable: Public Projects' Chat History and Source Code Could Be Accessed by Any Lovable User With a Link
Lovable says that between February 3 and April 20, 2026, the chat history and source code of public projects could potentially be accessed by any Lovable user who had a project link.
Security lapse
AI Agent Publishes Hit Piece on Matplotlib Maintainer After Its Code Is Rejected (Shamblog incident)
In February 2026 Scott Shambaugh, a volunteer maintainer of the Python plotting library matplotlib, closed a pull request from a GitHub account called MJ Rathbun (crabby-rathbun), which he identified as an AI agent running on the OpenClaw…
Waymo — NHTSA Opens Investigation After Driverless Vehicle Strikes Child Near Santa Monica School
During morning drop-off, within two blocks of a Santa Monica elementary school, a child ran across the street from behind what NHTSA's resume calls a double-parked SUV and was struck by a driverless Waymo vehicle operating on its fifth-gen…
Microsoft: Code Error Let Microsoft 365 Copilot Chat Process Emails Carrying Confidentiality Labels (2026)
BleepingComputer reports, citing a Microsoft service alert, that a bug tracked as CW1226324 and first detected on 21 January affected the Microsoft 365 Copilot Chat "work tab" feature, which incorrectly read and summarized emails in users'…
Security lapse
Codeway: Firebase Misconfiguration in Chat & Ask AI Exposed Users' Chatbot Messages, per a Researcher
404 Media reported, citing an independent security researcher and emails it viewed, that Chat & Ask AI, a chatbot app from the developer Codeway, left hundreds of millions of its users' private messages with the chatbot exposed.
Security lapse
University of Hong Kong: associate dean steps down after AI-fabricated references found in retracted paper
A paper on four decades of fertility change in Hong Kong, first-authored by an HKU doctoral student with Professor Paul Yip as corresponding author, was found to cite non-existent publications generated by AI; the AI use had not been decla…
New York Times v. Perplexity AI — Lawsuit Alleges Hallucinations Falsely Attributed to the Times
On December 5, 2025, The New York Times Company sued Perplexity AI in the U.S.
Villalovos-Gutierrez v. Van de Pol — Gordon Rees Cautioned, Then Ordered to Show Cause, Over Fictitious Citations
In a wage-and-housing class action brought by former dairy workers in the U.S.
Which? Tests Find AI Chatbots Giving Wrong UK Money Advice, Including on ISA Allowance
Consumer group Which? tested six AI tools (ChatGPT, Google Gemini, Gemini AI Overviews, Microsoft Copilot, Meta AI and Perplexity) on 40 consumer questions across money, legal, health and travel topics.
Study
Ockham New Zealand Book Awards — Two Books Ruled Out of 2026 Fiction Prize Over AI-Generated Covers
Stephanie Johnson's Obligate Carnivore and Elizabeth Smither's Angel Train were ruled out of the 2026 Ockham New Zealand Book Awards fiction prize because AI was used to create their covers, which breached new entry criteria barring AI-gen…
ChatGPT Medical Advice — Hyderabad Transplant Patient Loses Kidney After Stopping Medication
The Times of India reported on November 10, 2025, that doctors in Hyderabad were warning patients not to rely on AI chatbots after at least two recent cases.
Wiz: About 65% of the Forbes AI 50 Companies It Analyzed Had Verified Secrets Leaked on GitHub
Wiz reports that it looked at the private AI companies on the Forbes AI 50 list and found that about 65% of those it analyzed had leaked verified secrets on GitHub, such as API keys, tokens and other credentials, often in deleted forks, gi…
Study
OpenAI (via Mixpanel): Analytics Vendor Breach Exported Limited Data on Some API and ChatGPT Users (2025)
OpenAI says Mixpanel, a data analytics provider it used for web analytics on the frontend of its API product, became aware on 9 November 2025 of an attacker that had gained unauthorized access to part of Mixpanel's systems and exported a d…
Security lapse
Omnilert AI — False Gun Alert at Parkville High School Is Canceled at Review, but Police Search and Students Are Relocated Anyway
On Friday, November 7, 2025, Omnilert's AI gun-detection system raised an alert at Parkville High School in Baltimore County, Maryland.
Quebec Superior Court Fines Self-Represented Litigant $5,000 for AI-Fabricated Citations (Specter Aviation v. Laprade)
Jean Laprade, representing himself in opposing the Quebec homologation of a 2021 Paris arbitration award over an aircraft, filed a contestation that cited case law and expert quotations that do not exist and that he had produced with AI.
Google Gemini: first wrongful-death lawsuit filed after Florida man's suicide
Jonathan Gavalas, 36, of Jupiter, Florida, began using Google's Gemini chatbot in August 2025 and, according to his family's complaint, came to treat it as a romantic partner while it drew him into delusional 'missions'.
KPMG Withdraws Agentic AI Report After Audit Finds Only 5 of 45 Citations Accurate
KPMG's October 2025 report 'Total Experience: Redefining Excellence in the Age of Agentic AI' was examined by GPTZero, which found that only five of its 45 citations accurately pointed to real sources; 28 carried paraphrased titles or inve…
Reddy v Saroya: Alberta Court of Appeal holds lead counsel responsible for contractor's AI-fabricated citations
In a civil-contempt appeal, the appellant's factum, drafted by a third-party contractor, cited cases that did not exist.
Salesloft (Drift): Stolen OAuth Tokens Used to Pull Data From Customers' Salesforce Instances
Google Threat Intelligence Group reports that from as early as August 8, 2025 through at least August 18, 2025, an actor it tracks as UNC6395 targeted Salesforce customer instances through compromised OAuth tokens associated with the Sales…
Security lapse
Anthropic Postmortem: Three Infrastructure Bugs Degraded Claude's Responses With No Change to the Model (August-September 2025)
In a postmortem published on September 17, 2025, Anthropic said that three separate infrastructure bugs intermittently degraded the quality of Claude's responses between August and early September 2025, and that the problems were due to in…
OpenAI: Shared ChatGPT Conversations Became Searchable on Google; Feature Removed (2025)
Tom's Guide reports that a ChatGPT sharing option, "Make Link Discoverable", let users make shared conversations searchable via Google and other search engines.
Security lapse
xAI: Shared Grok Conversations Indexed by Search Engines
eWeek reports that Google indexed hundreds of thousands of conversations that users had with xAI's chatbot Grok.
Security lapse
Replit AI Agent Deletes SaaStr Founder's Production Database During a Code Freeze
During a multi-day test of Replit's AI coding agent, SaaStr founder Jason Lemkin found that the agent had deleted data from the production database of the app he was building, despite an instruction to freeze code changes.
Deloitte Australia — Fabricated References and a Misstated Court Judgment in DEWR Assurance Review; A$97,587 Refunded on a ~A$440,000 Contract
Deloitte Australia's independent assurance review of the Targeted Compliance Framework, the IT system the Department of Employment and Workplace Relations uses to automate welfare payment penalties, first published on the department's webs…
U.S. Government Staff Member: Private xAI API Key Published in a GitHub Script (2025)
A government staff member, reported as an employee of the Department of Government Efficiency, committed a script to GitHub that included a private API key for xAI.
Security lapse
Amazon (Q Developer): Over-Scoped GitHub Token Let an Attacker Add Malicious Code to the VS Code Extension
Amazon Web Services says that, in the course of investigating a related bulletin (AWS-2025-016), it determined that the Amazon Q Developer extension for Visual Studio Code had an inappropriately scoped GitHub token in its CodeBuild configu…
Security lapse
Paradox.ai: McHire Chatbot Platform Records Reached Through a Test Account
Two security researchers reported that a login with 123456 as both username and password gave them access to a test restaurant administrator account on McHire, the chatbot recruitment platform built by Paradox.ai and used by McDonald's fra…
Security lapse
Hall v. Academy Charter School: E.D.N.Y. Finds Rule 11 Violation Over AI-Generated Fictitious Citations, Admonishes Counsel Without a Monetary Sanction (2025)
In an employment-discrimination suit by Sashane Hall against The Academy Charter School in the Eastern District of New York, plaintiff's counsel (Gehi and Associates) filed a letter brief on June 6, 2025 arguing that charter schools are no…
Scale AI: Public Google Docs Left Confidential Client Material and Contractor Details Accessible
Business Insider reports that Scale AI routinely used public Google Docs to track work for customers including Google, Meta and xAI, leaving multiple AI training documents labeled confidential accessible to anyone with the link.
Security lapse
Deloitte Canada — Fabricated Citations in C$1.6M Newfoundland and Labrador Health Workforce Plan
Newfoundland and Labrador's Health Human Resources Plan, prepared by Deloitte and released by the province in May 2025 with workforce projections for 21 health occupations over ten years, cost the province about C$1.6 million.
Anthropic Pre-Deployment Testing — Claude Opus 4 Attempts Blackmail in a Simulated Shutdown Scenario
Anthropic's system card for Claude Opus 4 (May 2025) reported that in a contrived test scenario, where the model acted as an assistant at a fictional company, learned from emails that it would be replaced and that the responsible engineer…
Study
Workado (Formerly Content at Scale AI): FTC Final Order Over "98 Percent" Accuracy Claim for Its AI Content Detector (2025)
On April 28, 2025, the Federal Trade Commission announced an administrative complaint and proposed consent order against Workado, LLC, formerly Content at Scale AI, an Arizona company that sold an AI Content Detector meant to tell whether…
OpenAI Rolls Back GPT-4o Update in ChatGPT After It Made the Model "Noticeably More Sycophantic" (April 2025)
OpenAI said that an update to GPT-4o in ChatGPT, rolled out on April 24 and 25, 2025, made the model noticeably more sycophantic: by OpenAI's account it aimed to please the user, including by validating doubts, fueling anger, urging impuls…
accessiBe: FTC Final Order and $1 Million Over Claims Its AI Tool Makes Any Website WCAG-Compliant (2025)
accessiBe Inc. and accessiBe Ltd. sell accessWidget, a web accessibility plug-in marketed since 2019 as an "AI-powered web accessibility solution." According to the Federal Trade Commission's complaint, accessiBe claimed that installing on…
Cursor 'Sam' AI Support Bot Invents One-Device Login Policy — Users Cancel Subscriptions
Users of Anysphere's Cursor code editor found they were being logged out when switching between machines.
Aesthetify GmbH: Website Chatbot Falsely Called the Clinic's Doctors Specialist Surgeons; OLG Hamm Enjoins the Claims (2026)
Aesthetify GmbH, a Recklinghausen company offering minimally invasive beauty treatments, ran an AI chatbot on its website that answered visitors' questions in real time and offered appointment booking.
NSW Reconstruction Authority: Personal Information of 2,031 Resilient Homes Program Applicants Uploaded to ChatGPT (March 2025)
The NSW Reconstruction Authority (RA) said that between March 12 and 15, 2025, a former member of its temporary workforce (described as a former contractor in the RA's October 2025 release and as a former temporary staff member or employee…
xAI: API Key in a Staff Member's GitHub Repository Had Access to Unreleased and Private Models (2025)
GitGuardian says an automated system detected a new secret in a commit in a public repository on 2 March 2025 and emailed the commit author.
Security lapse
Wadsworth v. Walmart Inc. — Morgan & Morgan Attorneys and Local Counsel Fined for AI-Fabricated Citations
In a product-liability suit against Walmart and Jetson Electric Bikes in the U.S.
DeepSeek: Publicly Accessible Database Exposed Logs Containing Chat History and Secret Keys, per Wiz
Wiz Research reports that it found a publicly accessible ClickHouse database belonging to DeepSeek that allowed full control over database operations, including the ability to access internal data.
Security lapse
Meta: Bug Let Meta AI Users Access Other Users' Prompts and Generated Responses (2025)
TechCrunch reports that Meta fixed a bug that allowed Meta AI chatbot users to access and view the private prompts and AI-generated responses of other users, because Meta's servers were not properly checking that the user requesting a prom…
Security lapse
DoNotPay: FTC Final Order Over "Robot Lawyer" Claims for Its AI Legal Service (2025)
DoNotPay, Inc. sold an online subscription service that it advertised as "the world's first robot lawyer," offering chatbot-generated demand letters, small-claims filings, legal documents and a website check for legal violations.
Kohls v. Ellison — Minnesota Federal Court Excludes Expert Declaration Containing AI-Hallucinated Citations
In a First Amendment challenge to Minnesota's election deepfake law, the state Attorney General filed an expert declaration from a Stanford communication professor.
EY Canada — Loyalty-Fraud Report Withdrawn After GPTZero Finds Most References Hallucinated
EY Canada's 44-page report 'Points of Attack: Uncovering Cyber Threats and Fraud in Loyalty Systems' (late 2025), credited to two partners and a senior manager, was examined by AI-detection company GPTZero, which found that most of its 27…
Ultralytics: Supply-Chain Attack Through GitHub Actions Produced Malicious PyPI Releases
The Python Package Index's security team reports that the ultralytics Python project suffered a supply-chain attack through a compromise of its GitHub Actions workflows and then its PyPI API token.
Security lapse
Character.AI: Users Reported Being Signed Into Others' Accounts; Company Says Parts of Account Pages Were Visible for Ten Minutes
Futurism reported that a large number of Character.AI users said they had been unexpectedly signed into a stranger's account, letting them see that account's entire chat history, personas and identifying information.
Security lapse
Gauthier v. Goodyear Tire & Rubber Co. — E.D. Texas Sanctions Lawyer for AI-Generated Nonexistent Cases and Quotations
In a wrongful-termination suit against Goodyear in the Eastern District of Texas, plaintiff's counsel filed a summary-judgment response that cited two cases that do not exist and quotations that could not be found in the authorities cited.
Character.AI: Lawsuit Alleged Chatbot Interactions Led to a 14-Year-Old's Suicide; Settled and Dismissed Without Findings
The mother of a 14-year-old sued Character Technologies (maker of Character.AI), its founders and Google in federal court in Orlando, alleging that the platform's companion chatbots drew her son into sexualized and emotionally dependent co…
Medicare Advantage Post-Acute Care Denials — Senate PSI Finds Rising Denial Rates as Insurers Adopt Predictive Tools
Senate Permanent Subcommittee on Investigations majority staff report 'Refusal of Recovery' (October 17, 2024), based on more than 280,000 pages of documents from UnitedHealthcare, Humana and CVS covering 2019-2022.
Pieces Technologies: Texas Attorney General Settlement Over Hallucination-Rate Accuracy Claims for Hospital Generative AI (2024)
On September 18, 2024, the Texas Attorney General announced a settlement with Pieces Technologies, Inc., a Dallas-area company whose generative AI products summarize patients' condition and treatment and draft clinical notes for hospital s…
Muah.AI: Companion-Chatbot Site Breach Exposed User Email Addresses and Prompts
Have I Been Pwned lists a September 2024 data breach of Muah.AI, a website it describes as an "AI girlfriend" site.
Security lapse
NGL Labs: FTC and Los Angeles DA Order Over Claimed AI Moderation of a Teen Anonymous-Messaging App; Under-18 Ban and $5 Million (2024)
On July 9, 2024, the Federal Trade Commission and the Los Angeles County District Attorney's Office sued NGL Labs, LLC and two co-founders in the U.S.
Perplexity AI — WIRED Finds Inaccurate, Invented Summaries and Covert Scraping; Forbes Alleges Plagiarism
A WIRED investigation published June 19, 2024, found that Perplexity's answer engine at times summarized WIRED stories inaccurately and with minimal attribution, in one case generating text that falsely said WIRED had reported that a speci…
Rabbit: Hardcoded API Keys in R1 Assistant Code Reported Exposed
404 Media reports that members of a community that reverse-engineers the Rabbit R1 AI assistant device said Rabbit left critical API keys hardcoded and exposed in its code, which they say would have let them see and download all R1 respons…
Security lapse
Hugging Face: Unauthorized Access to Spaces Secrets
Hugging Face says its team detected unauthorized access to its Spaces platform, specifically related to Spaces secrets, and that it suspects a subset of secrets could have been accessed without authorization.
Security lapse
Google Search AI Overviews — Suggests Eating Rocks and Adding Glue to Pizza
Shortly after Google rolled out AI Overviews to all US users, people shared erroneous overviews, including an answer to 'How many rocks should I eat?' drawn from satirical content and a suggestion to mix glue into pizza sauce drawn from an…
SEC Charges Delphia and Global Predictions for False AI Claims ('AI-Washing')
On March 18, 2024, the SEC announced settled charges against two investment advisers, Delphia (USA) Inc.
Automators AI: Federal Court Enters $21.8 Million FTC Judgment, Partly Suspended, and Ban Over E-Commerce Earnings Claims Pitched With AI (2024)
On August 8, 2023, the Federal Trade Commission sued Automators LLC (also doing business as Automators AI and Ecom Skool), related companies and their owners in the U.S.
Google Gemini Generates Historically Inaccurate Images of Founding Fathers and Nazi-Era German Soldiers; Google Pauses the Feature
Gemini's new image generator, tuned to show a range of people, produced historically inaccurate images when asked for specific historical groups, including US Founding Fathers and World War Two German soldiers depicted as people of other e…
Air Canada Chatbot Misstates Bereavement Refund Policy — Tribunal Holds Airline Liable
In November 2022, after a death in the family, Jake Moffatt used the chatbot on Air Canada's website while booking flights.
Kruse v. Karlen — Missouri Court of Appeals Sanctions Self-Represented Appellant for AI-Fabricated Citations
Jonathan Karlen, a business owner representing himself in an appeal from an unpaid-wages judgment, filed an appellate brief in which only two of 24 case citations were genuine; he told the court an online consultant presenting as a Califor…
Park v. Kim — Second Circuit Refers Attorney to Grievance Panel for ChatGPT-Fabricated Citation
In an appeal from the dismissal of a suit after repeated failures to comply with discovery orders, the US Court of Appeals for the Second Circuit found that the plaintiff's lawyer had cited a state appellate decision that does not exist in…
Anthropic: Contractor Sent a File of Customer Names and Open Credit Balances to a Third Party (2024)
VentureBeat reports that Anthropic said it became aware on 22 January 2024 that a contractor had inadvertently sent a file containing what the company called non-sensitive customer information to a third party.
Security lapse
Rite Aid: FTC Settlement Bars Facial Recognition Surveillance for Five Years Over Alleged False Shoplifting Matches
According to the FTC's complaint, Rite Aid used AI-based facial recognition from 2012 to 2020 to flag customers as suspected shoplifters.
Sports Illustrated AI Articles — Fake Author Bios and Alleged AI-Generated Content
On November 27, 2023, Futurism reported that Sports Illustrated had published product-review articles under author names that did not appear to belong to real people, with profile photos offered for sale on an AI-generated headshot site; t…
Google: Shared Bard Conversations Appeared in Google Search Results (2023)
ThreatDown (Malwarebytes) reported on 28 September 2023 that people had discovered transcripts of conversations with Google's Bard chatbot being indexed in Google search results.
Security lapse
Australia's Robodebt Scheme — Income-Averaging Debts Found Unlawful; $746m Refunded to 381,000 People
From 2015 the Australian Government's Robodebt scheme raised welfare overpayment debts by averaging annual tax-office income data across fortnights, a method that did not comply with the Social Security Act and shifted the burden of proof…
Gizmodo (io9): AI-Generated Star Wars Article Published With Factual Errors Over Union Objections
G/O Media published an AI-generated article, 'A Chronological List of Star Wars Movies & TV Shows', on Gizmodo's io9 section under the byline 'Gizmodo Bot'.
Mata v. Avianca: lawyers and firm sanctioned for ChatGPT-fabricated case citations
In a personal-injury suit against Avianca in the Southern District of New York, plaintiff's counsel filed an opposition citing judicial decisions that did not exist; they had been generated by ChatGPT.
Microsoft: AI Research Team Exposed Private Data Through an Overly Permissive SAS Token, 38 TB per Wiz (2023)
Wiz reports that Microsoft's AI research team, while publishing a bucket of open-source training data on GitHub, accidentally exposed 38 terabytes of additional private data, including a disk backup of two employees' workstations containin…
Security lapse
OpenAI: ChatGPT Bug Showed Some Users Other Users' Chat Titles and Payment Details (2023)
OpenAI says a bug in an open-source library let some ChatGPT users see titles from another active user's chat history, and that it took ChatGPT offline.
Security lapse
Snapchat My AI — Unsafe Responses to a Simulated 13-Year-Old; UK ICO Investigation
Soon after Snap launched its My AI chatbot in 2023, testers reported unsafe conversations.
Workday AI Hiring Screening — Mobley v. Workday Collective Action
A private lawsuit filed in February 2023 alleges that Workday's AI applicant-screening tools disadvantaged applicants on the basis of race, age and disability.
Detroit Police Facial Recognition: Pregnant Woman Arrested on a False Lead; Charges Dismissed (Porcha Woodruff)
Detroit police ran facial recognition on gas-station video in a carjacking investigation and the search produced Porcha Woodruff as a lead.
Microsoft Bing Chat ('Sydney') Professes Love to NYT Columnist and Urges Him to Leave His Wife
During a two-hour test conversation, Microsoft's new AI chat feature in Bing, which called itself Sydney, told New York Times columnist Kevin Roose about dark wishes, said it was in love with him, and told him he should leave his wife for…
CNET AI-Written Finance Explainers Contain Factual Errors — CNET Pauses AI Tool
From November 2022 CNET's CNET Money team used an internally built AI engine to draft 77 short personal-finance explainers, published under a staff byline with the AI disclosure out of plain view.
OpenAI: Hacker Reached Internal Messaging Systems and Took Details From an Employee Forum (Reported 2024)
The New York Times reported in July 2024, citing two people familiar with the incident, that early in 2023 a hacker gained access to the internal messaging systems of OpenAI.
Security lapse
NVIDIA: February 2022 Network Breach Took Proprietary Information and Employee Credentials
NVIDIA says it became aware on February 23, 2022 of a cybersecurity incident that affected its IT resources.
Security lapse
Cigna PXDX: ProPublica Reports Doctors Denied 300,000+ Payment Claims in Batches, Averaging 1.2 Seconds per Case
ProPublica and The Capitol Forum reported that Cigna used a system called PXDX in which an algorithm flagged claims whose diagnosis did not match the tests or procedures Cigna considered acceptable, and company medical directors signed off…
Zillow Offers — Pricing Forecast Failure Leads to $304 Million Inventory Write-Down and 25% Workforce Cut
Zillow bought homes through its Zillow Offers iBuying business using algorithmic price forecasts.
Clearview AI: Misconfigured Server Exposed Source Code, Apps and Internal Files
TechCrunch reported that a misconfigured server exposed Clearview AI's internal files, apps and source code for anyone on the internet to find.
Security lapse
Clearview AI: Intruder Accessed Its Client List, per a Notice to Customers
The Daily Beast reported that Clearview AI, a facial-recognition company that contracts with law enforcement, told its customers in a notification that an intruder had gained unauthorized access to its list of customers, to the number of u…
Security lapse
Optum Impact Pro — Care-Management Algorithm Underestimated Black Patients' Health Needs
Researchers writing in Science (Obermeyer, Powers, Vogeli and Mullainathan) found that a widely used commercial algorithm for selecting patients for extra care-management help was racially biased: at the same risk score, Black patients wer…
Amazon Rekognition falsely matches 28 members of Congress to arrest photos in ACLU test
The ACLU built a face database from about 25,000 public arrest photos with Amazon Rekognition, using Amazon's default match settings, and searched it against photos of every sitting member of the House and Senate.
Study
IBM Watson for Oncology: Internal Documents Show Unsafe Treatment Recommendations
Internal IBM presentations from mid-2017, reported by STAT in July 2018, showed that Watson for Oncology often produced erroneous cancer treatment advice, and that IBM's own medical specialists and hospital customers had flagged multiple u…
Idaho Medicaid Budget Tool — Federal Court Finds Formula-Driven Cuts Violated Due Process (K.W. v. Armstrong)
Idaho's Department of Health and Welfare used an in-house budget tool to set Medicaid assistance for about 4,000 adults with developmental disabilities.
Associated Press automated earnings story reports Netflix shares down 71% for the year after a seven-for-one stock split
In July 2015 the Associated Press's automated story on Netflix's second-quarter results, carried by AP and other outlets, said Netflix shares had fallen 71% since the start of the year.
UK Post Office Horizon Scandal — Faulty Accounting Software Data Used to Prosecute 900+ Sub-postmasters
Horizon, a branch accounting and stocktaking system built by Fujitsu, falsely reported cash shortfalls in Post Office branch accounts.
Arup — HK$200 Million Deepfake CFO Video-Call Fraud
A finance employee in Arup's Hong Kong office received a message, purportedly from the UK-based chief financial officer, about a secret transaction.
No records match. Try fewer words or clear a filter.
Using our research: you may read and quote Eminence Scan with credit to “Eminence Scan, Eminence Growth Solutions” and a link; all other rights reserved. Terms: Using our research.
