Back to 2026 Abstracts

---

Accuracy and Inter-Rater Reliability of Clinician Decision-Making Before and After Exposure to an AI Hydronephrosis Severity Index
Samer Maher, MHSc¹, Adree Khondker, MD¹, Jin Kyu Kim, MD², Lauren Erdman, PhD³, Hadel Alsubaie, MD⁴, Laura Betcherman, MD⁴, Fabio Botelho, MD⁴, Valentina Bruno, MD⁴, Martina Bruneira, MD⁵, Joana Dos Santos, MD⁴, Usman Kahloon, MPH⁴, Jethro Kwong, MD¹, Mirriam Mikhail, MD⁴, Asmaa A. Milyani, MD⁴, Beverly Miranda, MN⁴, David-Dan Nguyen, MD¹, Innocent Nzeyimana, MD⁴, Mawuenyo Oyortey, MD⁴, Priya Saini, MD⁴, Nithiakishna Selvathesan, MD⁴, Chia W. Teoh, MD⁴, Marilyn Wong, MD⁴, Michael Chua, MD⁴, Armando Lorenzo, MD⁴, Mandy Rickard, MN⁴.
¹University of Toronto, Toronto, ON, Canada, ²Riley Hospital for Children, Indianapolis, IN, USA, ³Cincinnati Children's Hospital, Cincinnati, OH, USA, ⁴The Hospital for Sick Children, Toronto, ON, Canada, ⁵University of Padova, Padova, Italy.

Background
The Hydronephrosis Severity Index (HSI) is a convolutional neural network that assigns a continuous severity score to paediatric hydronephrosis from ultrasound images and has undergone multi-centre external validation. Validated discrimination alone does not establish clinical influence; a silent trial is the standard pre-deployment step to characterize how clinicians use a model and whether it changes management.
Methods
Twenty-three clinicians from paediatric urology and nephrology (residents, fellows, advanced practice providers [APPs], staff physicians) independently reviewed 293 retrospective, de-identified hydronephrosis cases (age 0-24 months), yielding 6,739 clinician-case observations. For each case, participants received standardized focused clinical information (e.g., postnatal age, laterality, prior imaging) and two ultrasound images, then selected a plan: discharge, repeat ultrasound, diuretic renogram, or surgery. They were then shown the HSI output (numerical risk score, predicted probability of surgery, traffic-light category) and recorded a revised or unchanged plan. The primary outcome, accuracy, was defined as concordance between a surgical (vs. non-surgical) recommendation and whether surgery was performed during follow-up (reference standard: real-world surgical decision). Pre- and post-model comparisons used McNemar’s test and a generalized linear mixed-effects logistic regression with random effects for clinician and case. Inter-rater reliability (ICC[2,1]) was compared via bootstrap.
Results
ICC(2,1) rose from 0.63 (95% CI 0.59-0.67) to 0.71 (0.68-0.75); Δ=0.08, p=0.0035. Decisions changed in 19.6% of clinician-case pairings (95% CI 18.7-20.6%); of these, the majority moved toward the HSI recommendation. Overall accuracy improved from 73.8% to 76.8% (Δ=+3.0 percentage points, McNemar p<0.001). In the mixed-effects model, model exposure was associated with increased odds of a correct decision (OR 1.44, 95% CI 1.28-1.62; p<0.001). Improvements were significant across all designations, but the magnitude differed by experience (interaction p=0.04): novice clinicians gained roughly twice the accuracy improvement of more experienced staff (Table 1).
Conclusion
The HSI model influenced clinician decision-making in one in five cases and produced significant improvements in accuracy and inter-rater reliability, with the largest gains among less-experienced clinicians. Because the reference standard is the real-world surgical decision rather than adjudicated need, and trainees in this trial were already within paediatric urology and nephrology, prospective workflow integration studies linked to patient outcomes are needed before clinical deployment.

Table 1. Decision Change and Accuracy Before and After Model Exposure by Clinician Designation.

Designation	Decision change rate (%)	Pre-model accuracy (%)	Post-model accuracy (%)	Δ accuracy (pp)	p-value
Resident/novice APP (n=7)	23.2	73.2	77.8	4.6	<0.001
Fellows (n=6)	19.9	73.4	76.0	2.6	<0.001
Staff physician/expert APP (n=10)	16.4	74.5	76.7	2.2	<0.001
Overall (n=23)	19.6	73.8	76.8	3.0	<0.001

---

Back to 2026 Abstracts