Generative AI Chatbots and Medical Misinformation — BMJ Open Audit Finds Half of Responses Problematic

Summary

A BMJ Open audit (Tiller et al., published April 14, 2026) put 50 health and medical questions to five consumer chatbots (Gemini, DeepSeek, Meta AI, ChatGPT and Grok) in February 2025, across cancer, vaccines, stem cells, nutrition and athletic performance. Of the 250 responses, 49.6% were rated problematic (30% somewhat and 19.6% highly). Grok produced more highly problematic responses than chance would predict and Gemini the fewest. Only two prompts (0.8%) were refused. Reference lists were poor (median completeness 40%), and hallucinated and fabricated citations meant no chatbot produced a fully accurate reference list. All readability scores fell in the 'difficult' band.

Who caught it A researcherFound outside the organization

This is a study: context for the failures library rather than an incident at one organization, so it carries no lesson.

Key dates

1. April 14, 2026Event
2. October 16, 2026Checked against sources
3. October 21, 2026Checked against sources

Developments are added when the record is reviewed. Next review: October 16, 2027.

Evidence

Each passage is quoted from its source. The quoted passages are re-checked against the live pages every month.

1. Nearly half (49.6%) of responses were problematic: 30% somewhat problematic and 19.6% highly problematic.

   Supports: Design (five chatbots, five domains, 250 responses), results, publication date 14 Apr 2026, DOI 10.1136/bmjopen-2025-112695

   Generative artificial intelligence-driven chatbots and medical misinformation: an accuracy, referencing and readability audit BMJ Open (via PubMed Central) Primary source

2. Chatbot hallucinations and fabricated citations precluded any chatbot from producing a fully accurate reference list.

   Supports: Fabricated-citation finding

   Generative artificial intelligence-driven chatbots and medical misinformation (abstract record) BMJ Open, via Europe PMC Primary source