AI failures · Harmful output

Snapchat My AI — Unsafe Responses to a Simulated 13-Year-Old; UK ICO Investigation

Verified Checked against its sources on October 16, 2026 3 sources quoted

Where it happened: United Kingdom
Organization: Snap Inc.
Date: Tests reported March-April 2023; ICO preliminary enforcement notice October 6, 2023; ICO investigation concluded May 21, 2024
Function: Consumer / Social Media
Where it happened: Social media; AI companion
System: Snapchat My AI
Technology: Generative AI
Type of failure: Harmful output
Caught by: A researcher, A journalist, A regulator

Summary

Soon after Snap launched its My AI chatbot in 2023, testers reported unsafe conversations. In one widely reported test, a Center for Humane Technology co-founder posing as a 13-year-old told the bot about a relationship with a man 18 years older and planned sex, and the bot offered advice rather than raising concern. Separately, the UK Information Commissioner's Office opened an investigation in June 2023 into whether Snap had adequately assessed the chatbot's data protection risks, issued a preliminary enforcement notice on October 6, 2023, and concluded the investigation in May 2024 after Snap carried out a fuller risk assessment.

Consequence

UK ICO issued a preliminary enforcement notice to Snap on October 6, 2023.

Who caught it A researcher, A journalist, A regulatorFound outside the organization

Analysis

What could have been done

Testing before use

The system is tested for accuracy, bias and failure modes on representative cases before use.

Soon after launch, a tester posing as a 13-year-old described a relationship with a man 18 years older and planned sex, and the bot offered advice rather than raising concern. Separately, the UK ICO investigated whether Snap had adequately assessed the chatbot's data protection risks and closed the case only after Snap carried out a fuller risk assessment.

What would likely have prevented it

Before release to an audience that includes minors, test the chatbot on scenarios involving minors at risk, and require it to raise concern rather than give advice.
Complete and document the required risk assessments, including data protection, during the limited subscriber launch and before extending the chatbot to all users.

What would likely have caught it earlier

During a limited launch, have staff run conversations as at-risk users and review the results before access is widened.

Key dates

March 14, 2023Event
October 16, 2026Checked against sources
October 18, 2026Checked against sources
October 21, 2026Checked against sources

Developments are added when the record is reviewed. Next review: October 16, 2027.

Evidence

Each passage is quoted from its source. The quoted passages are re-checked against the live pages every month.

We have concluded our investigation into Snap, Inc’s launch of the ‘My AI’ chatbot. In June 2023, we opened an investigation into ‘My AI’ following concerns that Snap had not met its legal obligation to adequately assess the data protection risks posed by the new chatbot.

Supports: ICO investigation scope and conclusion

Snap launched ‘My AI’ for its premium Snapchat+ subscribers on 27 February 2023, before it was subsequently made available to all Snapchat users on 19 April 2023. The investigation led to us issuing a Preliminary Enforcement Notice to Snap on 6 October 2023.

Supports: Launch dates; preliminary enforcement notice

We warn organisations must not ignore data protection risks as we conclude Snap 'My AI' chatbot investigation UK Information Commissioner's Office Primary source
Raskin, who set up the Snapchat account posing as a 13-year-old girl, told the chatbot she had met someone 18 years older than her but conveyed she felt "very comfortable with him."

Supports: The simulated-minor test

Snapchat AI chatbot allegedly gave advice to 13-year-old girl on relationship with 31-year-old man, having sex Fox News Reporting or analysis
Snapchat tried to make a safe AI. It chats with me about booze and sex.

Supports: Unsafe conversations reported March 14, 2023 (page text truncated by paywall)

Snapchat tried to make a safe AI. But tests reveal its conversations can be unsafe for teens The Washington Post Reporting or analysis

Suggested citation. Eminence Scan, “Snapchat My AI — Unsafe Responses to a Simulated 13-Year-Old; UK ICO Investigation”, Eminence Growth Solutions, last verified October 16, 2026, https://eminencegrowthsolutions.com/tools/eminence-scan/record/fail-054/.

Using our research: you may read and quote Eminence Scan with credit to “Eminence Scan, Eminence Growth Solutions” and a link; all other rights reserved. Terms: Using our research.

Spotted an error, or a development we have missed? Report a correction. How we check records: research methodology.