ProDef-MDS: A Proactive Defense Mechanism Protecting Malware Detection Systems from Adversarial Attacks

Malware threatens cybersecurity by enabling data theft, unauthorized access, and extortion. Traditional malware detection systems (MDS) struggle with the increasing volume and complexity of malware. While machine learning (ML) and deep learning (DL) offer promising solutions, they remain vulnerable to adversarial attacks that evade detection. Recent research focuses on developing adversarial datasets to retrain ML/DL-based malware detection systems, enhancing their robustness against adversarial attacks. While these methods improve detection of adversarial samples, they also cause more misclassification of non-adversarial data due to overfitting. These methods lack scalability when ML/DL-based MDS are retrained in isolation, without utilizing knowledge from other MDS with retrained models, leading to inefficiency and waste. To tackle these issues, we introduce ProDef-MDS, a proactive defense system that integrates an Adversarial Restoration (AR) module to mitigate adversarial perturbations and recover inputs to a correctly classifiable form before passing them into the malware classification model. We focus on portable executable (PE) malware within Windows OS to evaluate our approach’s effectiveness across various scenarios, including those with adversarial data generated from five white-box attacks, including Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD)10, PGD100, DeepFool, Carlini and Wagner (CW) and one black-box attack Auxiliary Classifier Generative Adversarial Networks (ACGAN). Additionally, we assess our approach with non-adversarial data to demonstrate its effectiveness in adversarial detection without compromising non-adversarial performance. The results obtained from the real-world dataset indicate enhanced robustness and minimal overhead, offering a proactive solution to adversarial threats in MDS. This approach outperforms retraining defense method in five white-box attacks and also indicates better performance in non-adversarial scenarios.