Objective Assessment and Standard Setting for Basic Flexible Ureterorenoscopy Skills Among Urology Trainees Using Simulation-Based Methods - Beyond the Abstract

The curricular reform occurring in the world of medical education centers around the concept of competence and pushes educators to ensure that residency produces graduates who meet a minimum standard of technical and non-technical ability. When relating these concepts to procedural skills, there are a number of ways in which this standard can be defined. One way that continues to grow in popularity amongst program directors and academics is the idea of ‘entrustability’, that is, the concept that newly qualified physicians and surgeons should be viewed as trustworthy in their execution of necessary clinical and surgical skills. In this study, we used a traditional standard setting method to determine entrustability standards for a core procedural skill amongst urologists, flexible ureteroscopy (fURS). We approached graduating urology residents and asked them to complete a series of straightforward, simulation-based ureteroscopic tasks, and scored their endoscopic skills using both a global rating scale (GRS) and a simple categorization based on how ‘entrustable’ they are based on said performance (entrustable, borderline, not entrustable). Ratings were completed by both expert surgeons and crowd-workers, using the CSATS© platform.

Through this process and using our data, we can begin to draw some conclusions. Firstly, it was apparent that despite these participants having completed at least four years of residency training at the time of testing, a seemingly high number were not thought to be entrustable in ureteroscopy. As stated in the manuscript, this could reflect a few things. It may be that in the intensive residency training curricula, these participants would not have in their final years of training completed as many ureteroscopies as an early-career urologist. Most residency programs in Canada use the final two years of training to develop trainees' open, laparoscopic, and robotic surgical skill sets, rather than endoscopic surgical procedures. This would explain why, in the setting of an isolated simulation-based assessment, so many of the residents failed to demonstrate entrustability. We speculate that perhaps training curricula should be designed to allow for graduating residents to have adequate exposure to these core surgical skills closer to the time of graduation.

Second, we note that the crowdsourced assessments and subsequently developed standards are different in a key manner from the expert-derived ones. Firstly, crowd workers were unwilling or unable to provide scores at either end of the GRS spectrum, instead of providing scores that deviate toward the mean of three-fifths, resulting in a narrow distribution of participant scores. While the aggregate scores are therefore not statistically different from the expert’s, this means that a small deviation in the decided ‘standard’ would result in a large change in the number of participants whose scores fall above and below the standard. Finally, it should be noted that these standards and therefore the number of participants that passed or failed the assessments is based on a traditional, participant-centered method, the ‘borderline method.’ This standard-setting approach does not account for actual clinical outcomes and relies entirely on the judgment of assessors to categorize participants based on their entrustability. Therefore, it is difficult to know whether those trainees that did not meet this standard would actually provide inadequate or unsafe care to patients in the practice, which ultimately should be the main focus of these benchmarking exercises.

This study provides us with valuable information about readiness for practice, the use of crowdsourced assessments for high-stakes decision making, and the use of entrustability as a concept to base technical skill assessments on. We are excited to follow this work up in the future with an examination of how these standards translate into the real world, and how we can use these methods to ensure graduating residents provide safe and competent care to patients across all facets of urology.

Written by: Mitchell G. Goldenberg, MBBS, PhD, Department of Surgery, University of Toronto, Toronto, Canada, and Jason Y. Lee, MD, MD, MHPE, FRCSC, Assistant Professor, Division of Urology, Department of Surgery, University Health Network - Toronto General Hospital, Urology Residency Program Director, Associate Member, Institute of Medical Science, University of Toronto, Toronto, Canada.

Read the Abstract