RSNA 2022: The Development of Pathology-Based Deep Learning Tools to Personalize Treatment Decisions in Radiation Oncology

( The 108th Scientific Assembly and Annual Meeting of the Radiological Society of North America (RSNA) held in Chicago, IL was host to a plenary session discussing machine learning in radiation oncology clinical trials and clinical practice. Dr. Felix Feng discussed the recent advance in the development of pathology-based deep learning tools to personalize treatment decisions.

Dr. Feng began his talk by noting that pathology-based artificial intelligence tools may allow for improvements in diagnosis (i.e. grading), prognosis (long-term outcomes), and prediction of treatment responses.


As a radiation oncologist with expertise in the field of genitourinary malignancies, Dr. Feng chose to highlight the recent advances for pathology-based machine learning in the field of prostate cancer. He began by noting that there are currently numerous NCCN-acknowledged prognostic biomarkers for prostate cancer patients with clinically localized disease. Importantly, there are currently no predictive biomarkers that are acknowledged by these guidelines.

risk strat.jpg

Recently, a multi-modal artificial intelligence (MMAI) prognostic tool for prostate cancer outcomes was developed and reported by Esteva et al. in Nature Digital Medicine. This MMAI tool utilized clinical data (PSA, Gleason score, age, stage) and digital pathology slides to develop an MMAI score. Importantly, this tool learns from clinical and histopathology data without slide annotations, lowering the potential barriers for usage and allowing for scalability.


This model was developed using data from five trials of prostate cancer patients with clinically localized disease receiving radiation therapy and having pre-treatment tissue biopsies (RTOG-9202, RTOG-9408, RTOG-9413, RTOG-9910, RTOG-0126). This included 5,654 patients and 16,204 slides total. Importantly, pooled data from these trials included a diverse patient population (African Americans ~20% of total cohort) and had long-term follow up (median follow up=11.4 years), which enhances the external validity and long-term accuracy, respectively, of this MMAI model.1

Following development of this MMAI score, the authors sought to evaluate whether this score could act as a prognostic biomarker. Of the 5,654 patients total, 80% were used to develop the score (development cohort) and the remaining 20% were correspondingly used as the validation cohort. Compared to the NCCN-acknowledged clinical data (combined Gleason score, T-stage, baseline PSA, Gleason primary, Gleason secondary, age), the MMAI score (using clinical and imaging data) was superior for prognosticating 5- and 10-year distant metastatic rates, 5- and 10-year biochemical failure rates, 10-year prostate cancer-specific survival, and 10-year overall survival:

mmai vs nccn.jpg

As such, this MMAI score has a clear role as a prognostic biomarker for patients with clinically localized prostate cancer. But the important question remained: can this score act as a predictive biomarker? Dr. Feng highlighted the differences between prognostic and predictive biomarkers:

  • Prognostic biomarkers provide information on outcomes independent of the treatment received
  • Predictive biomarkers specifically identify response or resistance to a particular therapy- but not all treatments

In this context, the authors sought to determine which patients receiving radiation therapy for clinically localized prostate cancer would benefit from concurrent hormonal therapy addition. In an analysis recently presented by Dr. Spratt at ASCO GU 2022, an MMAI predictive biomarker using data from the NRG Biobank (RTOG 9202, RTOG 9413, RTOG 9910, RTOG 0126) was developed using data from 3,935 patients. This MMAI score was developed used patient data, clinical data (Gleason score, T-stage, PSA), and digital pathology data. This score was modeled to predict distant metastases. After the model was locked, it was validated in the NRG/RTOG 9408 trial, which randomized 2,028 patients to radiotherapy plus short-term ADT or radiotherapy alone.

dev of mmai.jpg

This MMAI model predicted that the majority of patients with intermediate risk disease do not benefit from concurrent ADT addition to radiotherapy:

diff in scores.jpg

Significantly though, this MMAI model successfully predicted which patients were more likely to benefit from concurrent ADT addition, with patients having an MMAI score above the pre-defined cut off (i.e. biomarker positive) benefiting from short-term ADT addition (HR: 0.33, 95% CI: 0.19 to 0.57, p<0.001). Conversely, those who had an MMAI score below the pre-determined cut off (i.e. biomarker negative) did not experience improved distant metastatic outcomes with ADT addition (HR: 1.00, 95% CI: 0.63 to 1.56, p=0.98).

pos neg line.jpg

Dr. Feng concluded his presentation by highlighting important reasons why artificial intelligence tools remain promising biomarkers:

  1. Performance
    1. Predictive of response to specific therapies
    2. Initial tools developed from thousands of patients
    3. Artificial intelligence tools may improve over time
    4. Artificial intelligence tools account for heterogeneity within samples
  2. Turnaround time: Immediate in a digital world
  3. Access: Potentially easier to implement internationally

Presented by: Felix Feng, MD, Professor, Department of Radiation Oncology, Urology, and Medicine, University of California at San Francisco, San Francisco, CA

Written by: Rashid Sayyid, MD, MSc – Society of Urologic Oncology (SUO) Clinical Fellow at The University of Toronto, @rksayyid on Twitter during the 108th Radiological Society of North America (RSNA) Scientific Assembly and Annual Meeting, Nov. 27 - Dec. 1, 2022, Chicago, IL 


  1. Esteva A, et al. Prostate cancer therapy personalization via multi-modal deep learning on randomized phase III clinical trials. NPJ Digit Med. 2022;5(1):71.