Advancing Virtual Lab Immersion Through YOLO-Powered Gesture Recognition in Natural HCI

This study presents a gesture-based human-computer interaction (HCI) system for virtual engineering education, leveraging a modified YOLO-NAS Pose model for real-time hand gesture recognition. Unlike traditional input methods (mouse and keyboard) and commercially available virtual interfaces, this approach provides natural interaction within a screen-based electrical power lab simulation. The proposed system was benchmarked against MediaPipe, OpenPose, and CNN-based models, demonstrating superior inference speed (~2.35ms per frame on NVIDIA T4) with competitive accuracy (96.2%), making it highly suitable for real-time educational applications.