Virtual Speech Therapy Room: A Machine Learning-Based Neuro-Behavior Sensing Virtual Reality System for Aphasia Assessment and Treatment Through Multimodal Fusion

Aphasia is a common condition following brain injury, traditionally assessed and treated by speech therapists through manual evaluations and conventional language rehabilitation. However, these methods are time-consuming, reliant on professionals, and subject to subjective biases. This study aims to develop a virtual speech therapy room with an automated assessment model to assist clinicians in evaluation. It provides immersive virtual reality (VR) language training modules, combining analysis of physiological data to achieve the goal of smart healthcare. Twenty individuals with aphasia (IWA) and ten healthy participants were involved, with aphasia subjects randomly assigned to the experimental and control group A, and healthy participants forming control group B. Clinical scales, VR tasks, and neurobehavioral data were measured as needed. Statistical analysis confirmed that using virtual reality can enhance the effectiveness of aphasia treatment interventions. Utilizing virtual reality and behavioral sensing technology, significant differences were observed in the left frontal and occipital regions between IWA and healthy participants, aligning with clinical observations of impaired language and visual processing areas. The assessment model, established through these data, achieved an average classification accuracy of 97% in distinguishing between individuals with aphasia and healthy participants using multimodal fusion with repeated cross-validation, indicating its potential as an auxiliary tool for physician assessment and treatment.