(1111) Validation of Multidisciplinary Consensus-Based Pediatric Tracheal Intubation Skill Assessment

Introduction: Adverse tracheal intubation-associated events (TIAEs) and severe oxygen desaturation are common and associated with laryngoscopist tracheal intubation (TI) skill level. There is no assessment tool for TI skill with robust validity evidence. We hypothesized that a TI skill assessment tool using video recording would demonstrate high inter- and intra-rater reliability and that scores would be higher for TIs with lower difficulty as validity evidence.

Methods: A TI skill assessment tool using video recording was developed through modified Delphi method with 9 content experts with the final 9-item tool yielded expert rating consistency with Cronbach’s α=0.7. Thirty TI video recordings from the pediatric intensive care unit (PICU) with C-MAC standard blades were independently rated by 3 raters after initial training. TI difficulty level was categorized by indication and outcomes: Easy (TI for procedures such as MRI, 1 attempt, no TIAE or desaturation), Intermediate (non-procedural TI, 1 attempt, SpO2 < 80% or technical TIAE), or Difficult (non-procedural TI, >1 attempt, SpO2 < 70%, with technical TIAE or TI associated cardiac arrest). A randomly chosen subset of 10 TI videos were scored again by the same raters 6 months later. Psychometric analysis was based on generalizability theory.

Results: A fully crossed video by rater by item variance components analysis demonstrated variance was attributable to TI video (25%), followed by rater (20%) and associated video x rater (7%) and rater x item (10%) interactions. The model yielded a generalizability coefficient of 0.86. Generalizability analysis demonstrated the reliability of the total score was 0.92. D-study analysis based on a single rater yielded a reliability estimate of 0.79. Based on the total score, the inter-rater reliabilities were 0.84 (R1, R2), 0.80 (R1, R3) and 0.84 (R2, R3). Based on the total score, intra-rater reliabilities for the 3 raters were 0.95, 0.82, and 0.85. For all 3 raters, mean total scores were highest for easy TIs and lowest for difficult TIs.

Conclusions: The TI skill assessment tool demonstrates high inter- and intra-rater reliability. Given the high reliability coefficient, a single trained rater would be sufficient to assess each TI video. Scores were higher for TIs with lower difficulty, supporting validity evidence.