Information Systems Auditing of Psychiatric Diagnoses Using Context-Enriched QA-Based Language Models

The growing use of Artificial Intelligence (AI)-driven decision support systems (DSS) in healthcare requires effective auditing frameworks to ensure diagnostic accuracy and transparency. This study introduces a novel information system (IS) approach for psychiatric diagnosis auditing, transforming traditional multi-label classification into a question-answering (QA)-based framework enriched with diagnostic descriptions and explanations. Evaluating traditional classifiers, transformer models, Large Language Model (LLM), and the proposed Psychiatric QA model, results show that task transformation and context enrichment improve accuracy, with the QA model achieving the highest sum-F1 score and lowest precision fail rate. This study contributes to AI-driven IS, algorithmic auditing, and healthcare decision support, with managerial implications for electronic health record integration, insurance claim validation, and clinician workflow optimization. • A QA-based framework is proposed to audit psychiatric diagnoses using clinical notes. • The approach incorporates contextual enrichment with diagnostic criteria to improve transparency and interpretability. • A custom Multi-label Rouge metric is introduced to evaluate accuracy in multi-label diagnostic settings. • Experimental results show that transformer-based models outperform traditional ML in auditing psychiatric labels.