Computer · Vol. 5 No. 3 (2026) · Pages 42-56

Proactive Malware Detection and Behavioral Analysis through an AI-Enhanced Cryptographic Framework

Published

Authors

  • Hasanain Flayyih Hasan University of Wasit - College of Computer Science and Information Technology
  • Saif Hameed Abbood Department of Cybersecurity, College of Computer Science and Information Technology, Wasit University, IRAQ
  • Haza Nuzli Abdull Hamed

DOI https://doi.org/10.31185/wjps.1130

Downloads

Received May. 31, 2026 Revised Jun. 3, 2026 Accepted Jun. 12, 2026 Online Sep. 30, 2026

Abstract

Today's malware is polymorphic, metamorphic, file-less and signature evasive. Although the classifiers based on machine learning (ML) and deep learning (DL) have seen great improvements in their ability to accurately detect attacks, they remain susceptible to adversarial perturbations, model-extraction attacks, and training data leakage. One of the difficulties with an AI inference pipeline is the absence of a direct incorporation of cryptographic guarantees of integrity and confidentiality into the pipeline. In response to this, this paper introduces a hybrid scheme called Hybrid AI-Enhanced Cryptographic Framework (H-AIECF) that merges static and dynamic feature engineering on the EMBER2024 benchmark with a cryptographic envelope composed of AES-256-GCM (confidentiality), SHA-256 (integrity), and ECDSA-signed verdicts (authentication). The portable-executable (PE) features of EMBER2024 are extracted using chi-square selection and Principal Component Analysis (PCA) to train a baseline Random Forest (RF) and a CNN-LSTM hybrid. In the binary detection task, the proposed framework achieves 98.74% accuracy, 98.52% precision, 98.91% recall, 98.71% F1-score and 0.81% False Positive Rate (FPR) that are all better than those of a legacy signature scanner, tuned Random Forest and a standalone CNN–LSTM. All in all, the overhead of the cryptographic layer is still below 5% of the total inference latency, demonstrating that the behaviour-aware detection and end-to-end cryptographic protection are not conflicting and mutually reinforcing characteristics of a trustworthy pipeline for malware-defence.

Keywords

  • Malware Detection
  • Artificial Intelligence
  • Cryptography
  • Machine Learning
  • EMBER2024
Views: 106Downloads: 108

References

[1] J. Sun, Y. Liu, S. Zhao, and Y. Liu, "A review of deep learning-based malware detection techniques," Neurocomputing, vol. 598, art. 128010, 2024. doi: 10.1016/j.neucom.2024.128010. URL: https://doi.org/10.1016/j.neucom.2024.128010 DOI: https://doi.org/10.1016/j.neucom.2024.128010

[2] A. Bensaoud, J. Kalita, and M. Bensaoud, "A survey of malware detection using deep learning," Machine Learning with Applications, vol. 16, art. 100546, 2024. doi: 10.1016/j.mlwa.2024.100546. URL: https://doi.org/10.1016/j.mlwa.2024.100546 DOI: https://doi.org/10.1016/j.mlwa.2024.100546

[3] P. Maniriho, A. N. Mahmood, and M. J. M. Chowdhury, "A Survey of Recent Advances in Deep Learning Models for Detecting Malware in Desktop and Mobile Platforms," ACM Computing Surveys, vol. 56, no. 6, art. 145, 2024. doi: 10.1145/3638240. URL: https://dl.acm.org/doi/10.1145/3638240 DOI: https://doi.org/10.1145/3638240

[4] R. J. Joyce, G. Miller, P. Roth, R. Zak, E. Zaresky-Williams, H. Anderson, E. Raff, and J. Holt, "EMBER2024 — A Benchmark Dataset for Holistic Evaluation of Malware Classifiers," in Proc. 31st ACM SIGKDD Conf. Knowl. Discov. Data Min. (KDD), 2025. doi: 10.1145/3711896.3737431. URL: https://dl.acm.org/doi/10.1145/3711896.3737431 DOI: https://doi.org/10.1145/3711896.3737431

[5] L. Qian and L. Cong, "Channel Features and API Frequency-Based Transformer Model for Malware Identification," Sensors, vol. 24, no. 2, art. 580, 2024. doi: 10.3390/s24020580. URL: https://www.mdpi.com/1424-8220/24/2/580 DOI: https://doi.org/10.3390/s24020580

[6] D. Li, Q. Li, Y. Ye, and S. Xu, "PAD: Towards Principled Adversarial Malware Detection Against Evasion Attacks," IEEE Transactions on Dependable and Secure Computing, vol. 21, no. 2, pp. 920–936, 2024. doi: 10.1109/TDSC.2023.3265665. URL: https://ieeexplore.ieee.org/document/10097719 DOI: https://doi.org/10.1109/TDSC.2023.3265665

[7] M. AL-Essa, G. Andresini, A. Appice, and D. Malerba, "Adversarial Attacks with Defense Mechanisms on Convolutional Neural Networks and Recurrent Neural Networks for Malware Classification," Applied Sciences, vol. 14, no. 4, art. 1673, 2024. doi: 10.3390/app14041673. URL: https://www.mdpi.com/2076-3417/14/4/1673 DOI: https://doi.org/10.3390/app14041673

[8] Y. Wang, T. Sun, S. Li, X. Yuan, W. Ni, E. Hossain, and H. V. Poor, "Adversarial Attacks and Defenses in Machine Learning-Empowered Communication Systems and Networks: A Contemporary Survey," IEEE Communications Surveys & Tutorials, vol. 25, no. 4, pp. 2245–2298, 2023. doi: 10.1109/COMST.2023.3319492. URL: https://ieeexplore.ieee.org/document/10268461 DOI: https://doi.org/10.1109/COMST.2023.3319492

[9] C. Connors and D. Sarkar, "Machine Learning for Detecting Malware in PE Files," in Proc. 22nd IEEE Int. Conf. Mach. Learn. Appl. (ICMLA), Jacksonville, FL, USA, 2023, pp. 2194–2199. doi: 10.1109/ICMLA58977.2023.00331. URL: https://ieeexplore.ieee.org/document/10460035 DOI: https://doi.org/10.1109/ICMLA58977.2023.00331

[10] N. H. Saeed, A. A. Hamza, M. A. Sobh, and A. M. Bahaa-Eldin, "Efficient feature ranked hybrid framework for android IoT malware detection," Scientific Reports, vol. 16, art. 35238, 2026. doi: 10.1038/s41598-026-35238-6. URL: https://www.nature.com/articles/s41598-026-35238-6 DOI: https://doi.org/10.1038/s41598-026-35238-6

[11] K. Shaukat, S. Luo, and V. Varadharajan, "A novel deep learning-based approach for malware detection," Engineering Applications of Artificial Intelligence, vol. 122, art. 106030, 2023. doi: 10.1016/j.engappai.2023.106030. URL: https://doi.org/10.1016/j.engappai.2023.106030 DOI: https://doi.org/10.1016/j.engappai.2023.106030

[12] W. Guo, W. Du, X. Yang, J. Xue, Y. Wang, W. Han, and J. Hu, "MalHAPGNN: An Enhanced Call Graph-Based Malware Detection Framework Using Hierarchical Attention Pooling Graph Neural Network," Sensors, vol. 25, no. 2, art. 374, 2025. doi: 10.3390/s25020374. URL: https://www.mdpi.com/1424-8220/25/2/374 DOI: https://doi.org/10.3390/s25020374

[13] O. Polat, A. A. Ahmad, S. Oyucu, E. Algül, F. Doğan, and A. Aksöz, "Temporal-Spatial Feature Extraction in IoT-Based SCADA System Security: Hybrid CNN-LSTM and Attention-Based Architectures for Malware Classification and Attack Detection," IEEE Access, vol. 13, pp. 102109–102132, 2025. doi: 10.1109/ACCESS.2025.3577761. URL: https://ieeexplore.ieee.org/document/11038215 DOI: https://doi.org/10.1109/ACCESS.2025.3577761

[14] A. A. Almazroi and N. Ayub, "Deep learning hybridization for improved malware detection in smart Internet of Things," Scientific Reports, vol. 14, art. 7838, 2024. doi: 10.1038/s41598-024-57864-8. URL: https://www.nature.com/articles/s41598-024-57864-8 DOI: https://doi.org/10.1038/s41598-024-57864-8

[15] F. Deldar and M. Abadi, "Deep Learning for Zero-day Malware Detection and Classification: A Survey," ACM Computing Surveys, vol. 56, no. 2, art. 36, 2024. doi: 10.1145/3605775. URL: https://dl.acm.org/doi/10.1145/3605775 DOI: https://doi.org/10.1145/3605775

[16] T. Li, Y. Jiang, C. Lin, M. Obaidat, Y. Shen, and J. Ma, "MalAF: Malware Attack Foretelling from Run-Time Behavior Graph Sequence," IEEE Transactions on Dependable and Secure Computing, vol. 21, no. 4, pp. 1951–1966, 2024. doi: 10.1109/TDSC.2023.3298905. URL: https://ieeexplore.ieee.org/document/10198280 DOI: https://doi.org/10.1109/TDSC.2023.3298905

[17] A. Yu, J. Kang, J. Morris, E. Bertino, and D. Lin, "Fight Malware Like Malware: A New Defense Method Against Crypto Ransomware," IEEE Transactions on Dependable and Secure Computing, 2024. doi: 10.1109/TDSC.2024.3364209. URL: https://ieeexplore.ieee.org/document/10428048 DOI: https://doi.org/10.1109/TDSC.2024.3364209

[18] K. Begovic, A. Al-Ali, and Q. Malluhi, "Cryptographic ransomware encryption detection: Survey," Computers & Security, vol. 132, art. 103349, 2023. doi: 10.1016/j.cose.2023.103349. URL: https://doi.org/10.1016/j.cose.2023.103349 DOI: https://doi.org/10.1016/j.cose.2023.103349

[19] M. Maleki Esfahani, G. Andresini, A. Appice, and D. Malerba, "Evaluating Realistic Adversarial Attacks against Machine Learning Models for Windows PE Malware Detection," Future Internet, vol. 16, no. 5, art. 168, 2024. doi: 10.3390/fi16050168. URL: https://www.mdpi.com/1999-5903/16/5/168 DOI: https://doi.org/10.3390/fi16050168

[20] K. He, D. D. Kim, and M. R. Asghar, "Adversarial Machine Learning for Network Intrusion Detection Systems: A Comprehensive Survey," IEEE Communications Surveys & Tutorials, vol. 25, no. 1, pp. 538–566, 2023. doi: 10.1109/COMST.2022.3233793. URL: https://ieeexplore.ieee.org/document/9994719 DOI: https://doi.org/10.1109/COMST.2022.3233793

[21] C. Hong, "Recent advances of privacy-preserving machine learning based on (Fully) Homomorphic Encryption," Security and Safety, vol. 4, art. 2024012, 2025. doi: 10.1051/sands/2024012. URL: https://sands.edpsciences.org/articles/sands/abs/2025/01/sands20240021/sands20240021.html DOI: https://doi.org/10.1051/sands/2024012

[22] R. R. Irshad, S. Hussain, S. S. Sohail, A. S. Zamani, D. Ø. Madsen, A. A. Alattab, A. A. A. Ahmed, K. A. A. Norain, and O. A. S. Alsaiari, "IoT-Enabled Secure and Scalable Cloud Architecture for Multi-User Systems: A Hybrid Post-Quantum Cryptographic and Blockchain-Based Approach Toward a Trustworthy Cloud Computing," IEEE Access, vol. 11, pp. 105479–105498, 2023. doi: 10.1109/ACCESS.2023.3318755. URL: https://ieeexplore.ieee.org/document/10247105 DOI: https://doi.org/10.1109/ACCESS.2023.3318755

[23] A. Alqahtani, M. O. Ohemeng, and F. T. Sheldon, "An Intelligent Sensing Framework for Early Ransomware Detection Using MHSA-LSTM Machine Learning," Sensors, vol. 26, no. 3, art. 952, 2026. doi: 10.3390/s26030952. URL: https://www.mdpi.com/1424-8220/26/3/952 DOI: https://doi.org/10.3390/s26030952

[24] F. C. Onwuegbuche, A. Olaoluwa, A. D. Jurcut, and L. Pasquale, "MLRan: A Behavioural Dataset for Ransomware Analysis and Detection," 2025. arXiv:2505.18613. URL: https://arxiv.org/abs/2505.18613. https://doi.org/10.48550/arXiv.2505.18613

[25] H. H. Al-Khshali, M. Ilyas, F. Sohrab, and M. Gabbouj, "Malware Detection with Subspace Learning-Based One-Class Classification," IEEE Access, vol. 12, pp. 81017–81029, 2024. doi: 10.1109/ACCESS.2024.3409937. URL: https://ieeexplore.ieee.org/document/10547023 DOI: https://doi.org/10.1109/ACCESS.2024.3409937

[26] H. S. Anderson and P. Roth, "EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models," 2018. arXiv:1804.04637. URL: https://arxiv.org/abs/1804.04637. https://doi.org/10.48550/arXiv.1804.04637

[27] R. M. M. Akhtar and T. Feng, "Detection of Malware by Deep Learning as CNN-LSTM Machine Learning Techniques in Real Time," Symmetry, vol. 14, no. 11, art. 2308, 2022. doi: 10.3390/sym14112308. URL: https://www.mdpi.com/2073-8994/14/11/2308 DOI: https://doi.org/10.3390/sym14112308