To examine the performance of large language models (LLMs) for the analysis of adverse events (AEs) associated with a perirectal hydrogel spacer (SpaceOAR) prior to prostate radiation.
We queried the Food and Drug Administration's (FDA's) Manufacturer and User Facility Device Experience (MAUDE) database to extract reports related to "SpaceOAR".
Ninety-seven reports were initially manually abstracted to classify the problems associated with each event, subsequent modifications in radiation timing, and severity using the Common Terminology Criteria for Adverse Events (CTCAE) score. We compared the accuracy of 3 families of LLMs when compared to human abstraction. The highest performing LLM was then used to classify AEs based on all available MAUDE data for spaceOAR (n = 1,455) from January 2015 - December 2024.
The ability of LLMs to correctly identify the AE outcomes were aggregated into an overall score. The highest-performing model was GPT-4o, with an overall score of 4.96 (σ = 0.00526) compared to the human reviewers who had an overall score of 4.99 (σ =.216). When run on all 1,455 reports, GPT-4o revealed that the most common primary problems were malpositioned gel (58.7%), infection/inflammation/abscess (10.4%), fistula (7.1%), and rectal ulcer (4.7%). ICU level care and death were reported 0.1% and 0.3% of the time, respectively.
These findings highlight the potential for LLMs to automate the time-consuming process of tabulating device-related AEs. Reported serious AEs associated with spaceOAR underscore potential safety concerns, warranting dynamic ongoing surveillance and careful consideration when opting to implement hydrogel spacers.
Urology. 2026 Jan 19 [Epub ahead of print]
Nishan Sohoni, Nimit S Sohoni, Ryan A Sutherland, Vinaik M Sundaresan, Shayan Smani, Prasanna Ananth, John A Onofrey, Sanjay Aneja, Marcin Miszczyk, Ho-Joon Lee, Julia E Olivieri, Michael S Leapman
Department of Urology, Yale University, New Haven, CT, USA., Department of Pediatrics, Yale University, New Haven, CT, USA; Yale Cancer Outcomes, Public Policy and Effectiveness Research Center, New Haven, CT, USA., Department of Urology, Yale University, New Haven, CT, USA; Department of Radiology & Biomedical Imaging, Yale University, New Haven, CT, USA; Department of Biomedical Engineering, Yale University, New Haven, CT, USA., Department of Biomedical Engineering, Yale University, New Haven, CT, USA; Department of Therapeutic Radiology, Yale University, New Haven, CT, USA; Center for Outcomes Research and Evaluation, Yale University, New Haven, CT, USA., Department of Urology, Comprehensive Cancer Center, Medical University of Vienna, Vienna, Austria; Collegium Medicum Faculty of Medicine, WSB University, Dąbrowa Górnicza, Poland., Department of Genetics and Yale Center for Genome Analysis, Yale University, New Haven, CT, USA., Department of Computer Science, University of the Pacific, Stockton, CA, USA., Department of Urology, Yale University, New Haven, CT, USA. Electronic address: .
PubMed http://www.ncbi.nlm.nih.gov/pubmed/41565161