Evaluating Autonomy, Performance, and Feedback among Pediatric Urology Trainees at a Free-Standing Children’s Hospital
Christopher Jaeger, MD1, Kate Kraft, MD2, Benjamin Zendejas, MD, MSc1, Bartley Cilento, MD, MPH1.
1Boston Children's Hospital, Boston, MA, USA, 2University of Michigan, Ann Arbor, MI, USA.
BACKGROUND: Competency-based medical education compels the use of workplace assessment tools to routinely evaluate trainee performance for accurate, longitudinal tracking of trainee development. Our department previously used the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) to assess Pediatric Urology fellow operative performance beginning in 2018. We received a small number of assessments and were not able to feasibly include rotating Urology residents in the process. To address these limitations, our department implemented the Society for Improving and Measuring Professional Learning (SIMPL) smartphone-based application for evaluation of both rotating Urology residents and Pediatric Urology fellows. SIMPL is an innovative and widely used operative assessment platform with established validity and reliability evidence that can be easily incorporated into the busy surgical workflow. It permits assessment of trainees using three scales (autonomy, performance, and case complexity) for any case performed with a faculty member. It also offers the opportunity for faculty to give formative feedback to trainees through an optional dictation function. Additionally, trainees have the opportunity to rate themselves and compare to faculty ratings. Our intent with this study was to compare engagement with the O-SCORE and SIMPL and detail our experience with the rollout of SIMPL in our department situated within a free-standing children’s hospital.
METHODS: All Pediatric Urology faculty, Pediatric Urology fellows, and rotating Urology residents in our department underwent standardized rater training to understand the SIMPL rating system prior to participation in the trial that began in March of 2021. The trainee and the faculty member would independently complete the 3 question evaluation following each case in which a SIMPL evaluation was initiated by either party. A descriptive analysis was undertaken to compare engagement with O-SCORE and SIMPL. Univariate associations between clinical status (fellow or resident) and operative autonomy, performance, and case complexity were calculated using the Chi-squared test.
RESULTS: In the 3 academic years prior to implementing SIMPL, 53 O-SCORE evaluations were completed by 8 faculty for 4 fellows covering 15 unique procedures. In the first 3 months of SIMPL implementation, 201 SIMPL evaluations were completed by 9 faculty for 2 fellows and 7 residents covering 34 unique procedures. Trainees received a mean of 13.3 O-SCORE evaluations (s=1.5) compared to a mean of 22.3 SIMPL evaluations (s=15.6). Fellows reached meaningful autonomy for 83.5% of cases compared to the resident rate of 24.6% (p < 0.001). Fellows were more likely to be deemed practice ready (93.7%) than residents (33.6%, p < 0.001) despite overall case complexity being not significantly different between fellows and residents (p=0.69). Faculty dictated feedback in 58.2% of all SIMPL evaluations.
CONCLUSIONS: Within Pediatric Urology, this is the first implementation of the workplace operative performance assessment application called SIMPL in a free-standing children’s hospital. In just 3 months since implementing SIMPL, the volume of operative evaluations and the diversity of cases evaluated has increased while including both residents and fellows in the assessment system. Faculty ratings on SIMPL have demonstrated graduated autonomy and performance by training level as expected.
Back to 2021 Abstracts