Accuracy and reliability of MedBrain in assisting triage and diagnosis of common acute pediatric conditions in Ethiopia

Background: Childhood illnesses are a leading cause of morbidity and mortality in Sub-Saharan Africa, where healthcare infrastructure and trained personnel are limited. MedBrain, a digital decision support system (DDSS), aims to enhance pediatric emergency care by supporting mid-level healthcare workers in low-resource settings. Objective: To evaluate MedBrain’s triage and diagnostic performance among children presenting with common acute conditions to the emergency departments of two large hospitals in Ethiopia. Methods: A prospective observational diagnostic accuracy study was conducted between July 2024 and April 2025 at St. Paul’s Hospital Millennium Medical College and Alert Comprehensive Specialized Hospital. MedBrain’s triage and diagnostic performance were compared against healthcare professionals’ triage and pediatricians’ top presumptive diagnoses as gold standards. Performance metrics included accuracy, sensitivity (Sn), specificity (Sp), positive predictive value (PPV), negative predictive value (NPV), likelihood ratios (LR+ and LR–), and Cohen’s Kappa for reliability. Diagnostic performance was assessed for MedBrain’s top three ranked diagnoses (Top 1: highest probability diagnosis; Top 2: top two diagnoses; Top 3: top three diagnoses). Results: Of 1,204 patients screened, 274 were excluded for conditions not yet represented in MedBrain’s database (including malaria), leaving 930 participants. Of which, most were infants (33.8%) and children under 5 (31.9%), with pneumonia (20.4%) the most common diagnosis. MedBrain achieved 72.2% triage agreement, with 3.7% over-triage and 24.2% under-triage. Total diagnostic accuracy was 84.1% (Top 1), 91.5% (Top 2), and 93.3% (Top 3), with Sn of 93.3% and PPV of 100%. For prevalent conditions (pneumonia, acute bronchitis, late-onset neonatal sepsis, acute gastroenteritis, bronchiolitis, and meningitis), accuracy exceeded 97.4%, and Sp and PPV were consistently perfect. Sn increased from 73.0–98.6% (Top 1) to ≥90–100% (Top 2–3).NPV increased from 97.1–99.9% (Top 1) to 98.9–100% (Top 2) and 99.2–100% (Top 3). LR– improved from 0.014–0.270 (Top 1) to 0–0.100 (Top 2) and 0–0.079 (Top 3). Similarly, Cohen’s Kappa rose from 0.830–0.993 (Top 1) to 0.943–1.000 (Top 2) and 0.955–1.000 (Top 3). Diagnostic failures were rare, highest for late-onset neonatal sepsis (0.8%), bronchiolitis (0.5%), and pneumonia (0.4%), and none for gastroenteritis. Conclusion: MedBrain demonstrated high diagnostic accuracy and reliability in Ethiopian pediatric emergency settings. Under-triage and limited disease coverage remain challenges, warranting further validation with expanded disease libraries and in diverse settings.