Computer-aided segmentation on MRI for prostate radiotherapy, part II: Comparing human and computer observer populations and the influence of annotator variability on algorithm variability.

Comparing deep learning (DL) algorithms to human interobserver variability, one of the largest sources of noise in human-performed annotations, is necessary to inform the clinical application, use, and quality assurance of DL for prostate radiotherapy.

One hundred fourteen DL algorithms were developed on 295 prostate MRIs to segment the prostate, external urinary sphincter (EUS), seminal vesicles (SV), rectum, and bladder. Fifty prostate MRIs of 25 patients undergoing MRI-based low-dose-rate prostate brachytherapy were acquired as an independent test set. Groups of DL algorithms were created based on the loss functions used to train them, and the spatial entropy (SE) of their predictions on the 50 test MRIs was computed. Five human observers contoured the 50 test MRIs, and SE maps of their contours were compared with those of the groups of the DL algorithms. Additionally, similarity metrics were computed between DL algorithm predictions and consensus annotations of the 5 human observers' contours of the 50 test MRIs.

A DL algorithm yielded statistically significantly higher similarity metrics for the prostate than did the human observers (H) (prostate Matthew's correlation coefficient, DL vs. H: planning-0.931 vs. 0.903, p<0.001; postimplant-0.925 vs. 0.892, p<0.001); the same was true for the 4 organs at risk. The SE maps revealed that the DL algorithms and human annotators were most variable in similar anatomical regions: the prostate-EUS, prostate-SV, prostate-rectum, and prostate-bladder junctions.

Annotation quality is an important consideration when developing, evaluating, and using DL algorithms clinically.

Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology. 2021 Dec 31 [Epub ahead of print]

Jeremiah W Sanders, Henry Mok, Alexander N Hanania, Aradhana M Venkatesan, Chad Tang, Teresa L Bruno, Howard D Thames, Rajat J Kudchadker, Steven J Frank

Department of Imaging Physics, The University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd., Unit 1472, Houston, TX, 77030, United States. Electronic address: ., Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd., Unit 1422, Houston, TX, 77030, United States., Department of Radiation Oncology, Baylor College of Medicine, 7200 Cambridge St, BCM 711, Houston, TX, 77030, United States., Department of Diagnostic Radiology, The University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd., Unit 1473, Houston, TX, 77030, United States., Department of Biostatistics, The University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd., Unit 1411, Houston, TX, 77030, United States. Electronic address: ., Department of Radiation Physics, The University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd., Unit 1420, Houston, TX, 77030, United States.

email news signup