RSNA 2022: The Hierarchical Relationships Between Deep Learning, Machine Learning, and Artificial Intelligence

( The 108th Scientific Assembly and Annual Meeting of the Radiological Society of North America (RSNA) held in Chicago, IL was host to a plenary session discussing machine learning in radiation oncology clinical trials and clinical practice with Dr. Quynh Thu Le presenting the latest updates in this field. Dr. Le began her presentation by highlighting the hierarchical relationships between deep learning, machine learning, and artificial intelligence:

  1. Artificial intelligence: Techniques that enable computers to mimic human intelligence
    • Machine learning: A subset of artificial intelligence that uses complex statistical algorithms to improve performing tasks with increasing experience
      • Deep learning: A subset of machine learning that allows software to train itself to perform tasks through multilayered neural networks


Artificial intelligence can be used within the context of clinical trials with regards to the following four domains:

  1. Eligibility and accrual
    • Optimize eligibility criteria
    • Match patients to trials
    • Address healthy equity in trial entry/enrollment
  2. Treatment
    • Surgical quality
    • Radiation oncology workflow and quality
  3. Outcome assessment
    • Risk mitigation of toxicity
    • Response assessment with imaging
  4. Prognostic models for future trial development or personalization of therapy via:
    • Medical records
    • Imaging
    • Digital pathology
    • Genomics 

How can artificial intelligence optimize eligibility criteria using real world data? Using data from the non-small cell lung cancer (NSCLC) disease space, Liu et al. used the Flatiron Health electronic health record (HER) database of 61,000 patients with advanced/metastatic NSCLC. This project was driven by the fact that 80% of patients with advanced NSCLC do not meet criteria for clinical trials and 86% of NSCLC trials fail to complete recruitment on time. The authors identified 10 NSCLC randomized trials with available protocols and greater than 250 patients. They correspondingly identified patients in the Flatiron database who had taken the treatment of protocol drug (~5000 patients per trial). Approximately 30% of the patients taking the drugs would have qualified for the trials based on original criteria. They next used the Applied Trial Pathfinder that integrates real-world data and systematically analyses the hazard ratios of the overall survival for cohorts that are defined by different eligibility criteria. They hypothesized that Trial Pathfinder would remove unneeded eligibility criteria, increase the number of eligible patients, while maintaining or improving the benefit of treatment. As demonstrated in the table below, use of data-driven criteria, as compared to use of original trial criteria, would significantly reduce the number of criteria in each trial, while increasing the number of eligible patients at a consistent treatment effect, quantified via the hazard ratio.1

RSNA Trial comparisons.jpg

Data driven criteria led to removal of approximately nine inclusion/exclusion criteria, more than doubled the number of eligible patients (from 1,553 to 3,209), improved the hazard ratio for overall survival by ~0.05, and increased the number of eligible women and elderly patients in the trials. This model has since been validated against progression-free survival, in more modern patients (2017-2020), and against smaller number of trials in melanoma, colorectal, and breast cancers. Importantly, these relaxed criteria were not associated with more toxicity. This model is currently being applied in prospective trials.

Another important unmet research/clinical need is the ability to integrate available clinical trials into a single resource to improve patient selection and enrolment into the most appropriate clinical trial, irrespective of the clinical center/setting the patient initially presents to. mCODE (minimal Common Oncology Data Elements) is a step towards capturing research-quality data from the treatment of all cancer patients. It is a Fast Healthcare Interoperability Resources (FHIR)-based core set of common data elements for cancer that is standardized, computable, clinically applicable, and available in every electronic health record.



Affected by.jpg

This system allows for integrated trial matching for cancer patients and providers by addressing the main problem of patients not being made aware of clinical trial opportunities outside of their treating institution. As such, this system creates an integrated, automated, site-agnostic clinical trial matching by developing open data standards and applicable programming interfaces that enable accessible clinical trial matching services. As such, patients and providers can easily identify potentially lifesaving therapies much faster with tools that leverage structured data from the electronic health records. Researchers can also find more patients for their clinical trial using this model.


This system may potentially be of most benefit in underserved communities where patients still suffer from lack of access to clinical trial enrollment, and racial/socioeconomic barriers to equitable care remain a significant concern.

Testing rates.jpg  

In the field of radiation oncology, artificial intelligence can have numerous applications:

  1. Treatment decision: Decision support tool that combines clinical, genomic, and imaging data to support precision oncology practices
  2. Imaging (simulation) planning: To reduce radiation exposure, enhance imaging quality, suppress artifacts, and enable more accurate image registration
  3. Treatment planning: For automated tumor and organ segmentation as well as optimal dose prediction to streamline the planning process
  4. Plan approval and quality assurance: To help expedite the quality assurance process and detect rare erroneous events, especially for highly complex treatments
  5. Radiotherapy delivery: Enhance image guidance, motion management, and scheduling to improve clinical efficiency and patients’ outcomes and experiences
  6. Follow-up care: To accurately predict response to treatment, radiation-induced toxicities, and other adverse effects that might provide real-time meaningful clinical decision support

AI in radiation.jpg 

Dr. Le however did note that model deterioration over time is one important pitfall of artificial intelligence models. A case example being prostate segmentation model deterioration over time A root cause analysis of this issue demonstrated that introduction of rectum spacers and magnetic resonance simulation for SBRT patients led to longitudinal changes that caused this deterioration. As such, a potential solution to this would be continued performance monitoring and future model updating.

pros rec line grph.jpg 

Dr. Le concluded her presentation as follows:

  • Artificial intelligence has a promising role in clinical trials and radiation oncology
  • The current applications of artificial intelligence are primarily in improving workflow, decision support tools, and clinical trials enrollment
  • As artificial intelligence relies on large volumes of data, areas of data deficiencies in underserved populations needs to be addressed
  • With more data and computer power, artificial intelligence algorithms will need continuing updates, which makes it hard to test them in prospective clinical trials using the traditional designs
  • Novel clinical trial designs, organizational frameworks and collaboration are needed to leverage the full power of artificial intelligence

Presented by: Quynh-Thu X. Le, MD, Professor and Chair, Department of Radiation Oncology, Stanford, Palo Alto, CA

Written by: Rashid Sayyid, MD, MSc – Urology Chief Resident, Augusta University/Medical College of Georgia, @rksayyid on Twitter during the 108th Radiological Society of North America (RSNA) Scientific Assembly and Annual Meeting, Nov. 27 - Dec. 1, 2022, Chicago, IL 


  1. Liu R, et al. Evaluating eligibility criteria of oncology trials using real-world data and AI. Nature. 2021;592:629-33.