Obfuscated malware detection using a hybrid of CNN and GRU models

The growing sophistication of cyber threats demands advanced and reliable malware detection methods. This study presents a hybrid deep learning framework for detecting obfuscated malware that combines convolutional neural networks (CNN) and gated recurrent units (GRU). The approach integrates an optimized feature selection pipeline using Pearson correlation analysis, recursive feature elimination with cross-validation (RFECV), univariate statistical tests, and permutation importance to obtain a compact and informative subset of 27 features from 55 processed memory-based attributes, including API call sequences and allocation patterns. The model employs three one-dimensional convolutional layers for spatial extraction, two GRU layers for temporal modeling, and dense layers for final classification. Experiments on the CIC-MalMem2022 dataset achieve training and validation accuracies above 99.9% and stable performance across multiple runs, with a mean test accuracy of 99.962% ± 0.008%. To test transferability, the trained models were evaluated on the external EMBER benchmark dataset and achieved 92.10% accuracy, confirming transfer beyond the source dataset. Anti-leakage safeguards were applied throughout, with RFECV and standardization confined to training folds to prevent information leakage into validation or test splits. Kernel SHAP analyses provided both global and local interpretability of model decisions. With a lightweight 1.37 MB footprint and a 5–10% improvement in detection rate over baseline hybrid architectures, the proposed CNN–GRU model demonstrates robust performance, reliable generalization, and clear interpretability, offering a practical and transparent solution for detecting sophisticated obfuscated malware in modern cybersecurity systems.