EAVS-TF: A bio-inspired spiking neural network for energy-efficient multimodal emotion recognition

Background and purpose Accurate recognition of emotional states is critical for monitoring neuropsychiatric disorders, however, deploying such systems on battery-powered wearable devices necessitates exceptional energy efficiency. This study aims to develop a lightweight, bio-inspired framework for energy-efficient multimodal emotion recognition. Methods We propose EAVS-TF, a Spiking Neural Network (SNN) framework integrating three key components: (1) a biologically motivated adaptive feature selection module that utilizes short-term leaky integration and sparse gating to dynamically recalibrate audio-visual channels; (2) a compact Transformer encoder for deep cross-modal fusion; and (3) an energy-efficient Leaky Integrate-and-Fire (LIF)-based spiking classifier. The model was evaluated using the CREMA-D benchmark dataset. Results EAVS-TF achieved a recognition accuracy of 81.48%, outperforming previous SNN-based methods by 3–11% points. Ablation studies confirmed that the adaptive feature selection mechanism significantly enhanced both robustness and fusion quality. Conclusions EAVS-TF successfully synthesizes the event-driven efficiency of SNNs with the representational power of Transformers. The framework demonstrates significant potential as a low-power, non-invasive tool for continuous behavioral monitoring in future wearable and neuromorphic systems