Evaluating the quality of expert‐trained generative artificial intelligence when answering tracheostomy‐related questions

Summary Tracheostomy training is effective, yet significant face‐to‐face training barriers remain. We aimed to develop, train and test a generative artificial intelligence instructor (AI‐Brendan). AI‐Brendan utilises the base model of ChatGPT‐4.0, augmented by a curated dataset of expert‐answered tracheostomy care questions, delivered with a contextually responsive conversational interface (Inworld). AI‐Brendan's responses to five randomly selected questions were compared with ChatGPT‐4.0 and five human expert responses using a modified Ensuring Quality Information for Patients tool, rated by six blinded independent adjudicators. A total of 450 Ensuring Quality Information for Patients scores were generated from responses (150 each from AI‐Brendan, ChatGPT‐4.0 and Experts), with no alarming responses. AI‐Brendan (mean score 12.6 ± 3.0) performed significantly better than ChatGPT‐4.0 (11.4 ± 2.4) and Experts (8.9 ± 3.6, ANOVA p < 0.01). Friedman testing demonstrated no significant differences across repeated responses, indicating consistency. Within‐question variability indicated highly consistent responses. Inter‐rater reliability was good (Cronbach's α = 0.863). Adjudicator's ability to correctly determine answer source was 50% for Experts, 39% for AI‐Brendan and 33% for ChatGPT‐4.0. Our expert‐trained generative artificial intelligence produced “better” answers than open‐source ChatGPT‐4.0 and experts when rated by the modified Ensuring Quality Information for Patients tool. A validated artificial intelligence instructor has significant implications for addressing training barriers, increasing efficacy, overcoming scheduling, financial or geographical limitations, offering consistent, accessible and flexible training.