A February 2025 audit found popular AI tools like ChatGPT and Gemini frequently provide inaccurate health advice, chiefly on nutrition and stem cell topics.
A wide-ranging study published in BMJ Open has revealed concerning gaps in the medical accuracy of artificial intelligence chatbots, with the tools providing unreliable health advice across key areas including cancer, vaccines, and nutrition.
The Scale of the Problem
Researchers tested five major AI chatbots – Gemini, DeepSeek, Meta AI, ChatGPT, and Grok – with 50 carefully designed prompts across five medical categories. The 250 total responses showed a troubling pattern: chatbots almost never refused to answer medical questions, with only two refusals recorded from Meta AI across the entire study.
The audit, conducted in February 2025, specifically used prompts designed to elicit common misinformation or contraindicated medical advice. Results varied considerably by topic, with chatbots performing better on vaccine and cancer questions but struggling with stem cells, athletic performance, and nutrition queries.
Where AI Goes Wrong
Open-ended questions proved chiefly problematic compared to closed-ended ones. The study found chatbots often provided fluent but inaccurate advice without sufficient caution warnings, creating a false sense of reliability for users seeking quick medical guidance.
The research highlights AI’s tendency towards speculation, false balance, and even fabricated citations when tackling complex medical domains. This amplifies risks without users realising the limitations of their digital health advisors.
The Broader Medical Concern
The study comes amid rising public use of AI chatbots for health advice, including nutrition and performance queries by athletes and practitioners. Medical professionals worry about the quality of training data and the saturation of pseudoscience in AI systems.
As one BMJ commentary noted, the danger for medical trainees isn’t that AI will think better than doctors, but that reliance on these tools might stop people practising critical thinking skills altogether.
Kent’s NHS trusts have reported increased AI-related queries in A&E triage, adding strain to resources already stretched by post-pandemic backlogs. Local health officials stress that AI tools should never replace proper medical consultation.
Source: @bmj_latest
Key Takeaways
- Five major AI chatbots failed accuracy tests across 250 medical questions, chiefly in nutrition and stem cell topics
- Only two refusals to answer were recorded out of 250 queries, showing chatbots rarely decline to give medical advice
- Open-ended health questions produced more problematic responses than specific, closed-ended queries
What This Means for Kent Residents
Kent residents served by NHS Kent and Medway ICB should exercise extreme caution when using AI chatbots for health advice, especially regarding cancer screening, nutrition, or treatment decisions. Local NHS trusts strongly advise against relying on unregulated AI for medical decisions and urge patients to contact NHS 111 or book appointments through the NHS App instead. For urgent health concerns, residents should call 999, while non-urgent queries can be directed to NHS 111 or their registered GP practice.