In 2024, 17% of adults reported using an artificial intelligence (AI) chatbot at least once a month as a source of health information, rising to 25% among those under 30. We aim to conduct a scoping review of the existing literature assessing the performance of large language model (LLM) chatbots for patient education in kidney stone disease (KSD).
The Joanna Briggs Institute methodology was followed. Ovid MEDLINE, Embase, CENTRAL, Web of Science, CINAHL, and Google Scholar were searched for studies in all languages, from 2015 up to February 16th 2025, evaluating LLM chatbots to create educational content on KSD. Two independent reviewers completed screening, full-text review, and data extraction, with conflicts resolved by a third reviewer.
Of the 281 search results, 17 were included. Five of six studies assessing readability found that LLM responses exceeded the recommended 6th-8th grade reading level, though effective prompting can help meet this target. Six out of eight studies reported adequate to very good accuracy performance, with two showing comparable performance to traditional information sources. Understandability and actionability performance was poor. Quality performance was variable across seven studies. Three studies assessed patients' perception, revealing a mixed but generally favorable experience. Two studies noted notable deviations from established clinical guidelines.
LLM chatbots show potential for KSD patient education and physician workload reduction, but currently have limitations in readability, guideline adherence, understandability, and actionability. This could be mitigated by prompting and the development of urology-specific tools trained on validated content and evaluated with patient involvement.
World journal of urology. 2025 Oct 29*** epublish ***
Reda Goudrar, Othmane Zekraoui, Ibrahim Moussa, David-Dan Nguyen, David Bouhadana, Tiange Li, Vineet Gauhar, Steffi Kar Kei Yuen, Naeem Bhojani
Faculty of Medicine, University of Montreal, Montreal, QC, Canada., Division of Urology, University of Toronto, Toronto, ON, Canada., Division of Urology, McGill University, Montreal, QC, Canada., Department of Urology, Ng Teng Fong Hospital, Singapore, Singapore., Department of Surgery, SH Ho Urology Centre, The Chinese University of Hong Kong, Shatin, Hong Kong, China., Division of Urology, University of Montreal Hospital Center, 900, rue Saint-Denis, pavillon R, Montréal, QC, H2X 0A9, Canada. .