Patients rely on online searches for patient education materials (PEMs). PEMs are recommended to be written at or below a sixth-grade reading level but are regularly written at a college reading level. Using prompt engineering, we assess the information, misinformation, and readability of ChatGPT responses to urologic oncology questions.
Forty-five questions relating to prostate, bladder, and kidney cancer were presented to ChatGPT (version 4o, OpenAI). Quality of health information was assessed using DISCERN (1 [low] to 5 [high]). Understandability and actionability were assessed using PEMAT-P (0 [low] - 100% [high]). Misinformation was scored from 1 [no misinformation] to 5 [high misinformation]. Grade and reading level were calculated using the Flesch-Kincaid scale [5 (easy) to 16 (difficult), and 100-90 (5th grade level) to 10-0 (professional level), respectively]. Prompt engineering was then applied to responses and evaluated.
ChatGPT answers are highly accurate but too advanced of a reading level and lacked explanations of benefits, risks, visual aids, actionability, and citations. Using prompt engineering, DISCERN (3.42-4.47, p<0.0001), PEMAT-P understandability (88.4-95.5%, p<0.0001), and actionability (25.6-84.2%, p<0.0001), grade reading level (10.5-5.3, p<0.0001), and reading level (42 [college level] to 71.7 [7th grade], p<0.0001), all significantly improved. Misinformation did not change significantly.
Using prompt engineering, ChatGPT provides highly accurate and understandable PEMs at a patient appropriate reading level and provides concrete resources for patient action. Urologists should understand prompt engineering and be involved in the development of artificial chatbots to optimize results.
Canadian Urological Association journal = Journal de l'Association des urologues du Canada. 2026 Mar 30 [Epub ahead of print]
Mark N Alshak, Michelle I Higgins, Craig Cronin, William S Azar, Joseph G Cheaib, Max Kates, Sunil H Patel
The James Buchanan Brady Urological Institute, Johns Hopkins University School of Medicine, Baltimore, MD, United States.