Grand challenges stimulate advances within the medical imaging research community; within a competitive yet friendly environment, they allow for a direct comparison of algorithms through a well-defined, centralized infrastructure. The tasks of the two-part PROSTATEx Challenges (the PROSTATEx Challenge and the PROSTATEx-2 Challenge) are (1) the computerized classification of clinically significant prostate lesions and (2) the computerized determination of Gleason Grade Group in prostate cancer, both based on multiparametric magnetic resonance images. The challenges incorporate well-vetted cases for training and testing, a centralized performance assessment process to evaluate results, and an established infrastructure for case dissemination, communication, and result submission. In the PROSTATEx Challenge, 32 groups apply their computerized methods (71 methods total) to 208 prostate lesions in the test set. The area under the receiver operating characteristic curve for these methods in the task of differentiating between lesions that are and are not clinically significant ranged from 0.45 to 0.87; statistically significant differences in performance among the top-performing methods, however, are not observed. In the PROSTATEx-2 Challenge, 21 groups apply their computerized methods (43 methods total) to 70 prostate lesions in the test set. When compared with the reference standard, the quadratic-weighted kappa values for these methods in the task of assigning a five-point Gleason Grade Group to each lesion range from - 0.24 to 0.27; superiority to random guessing can be established for only two methods. When approached with a sense of commitment and scientific rigor, challenges foster interest in the designated task and encourage innovation in the field.
Journal of medical imaging (Bellingham, Wash.). 2018 Nov 10 [Epub]
Samuel G Armato, Henkjan Huisman, Karen Drukker, Lubomir Hadjiiski, Justin S Kirby, Nicholas Petrick, George Redmond, Maryellen L Giger, Kenny Cha, Artem Mamonov, Jayashree Kalpathy-Cramer, Keyvan Farahani
The University of Chicago, Department of Radiology, Chicago, Illinois, United States., Radboud University Medical Center, Department of Radiology and Nuclear Medicine, Nijmegen, The Netherlands., University of Michigan, Department of Radiology, Ann Arbor, Michigan, United States., Frederick National Laboratory for Cancer Research, Cancer Imaging Program, Frederick, Maryland, United States., U.S. Food and Drug Administration, Center for Devices and Radiological Health, Silver Spring, Maryland, United States., National Cancer Institute, Cancer Imaging Program, Division of Cancer Treatment and Diagnosis, Bethesda, Maryland, United States., MGH/Harvard Medical School, Boston, Massachusetts, United States.