A study on functionality validation for windows malware mutating using reinforcement learning

To keep pace with the rapid advancements in both the quality and complexity of malware, recent research has extensively employed machine learning (ML) and deep learning (DL) models to detect malicious software, particularly in the widely used Windows system. Despite demonstrating promising accuracy in identifying malware, these models remain vulnerable to adversarial attacks, where carefully modified malware samples can bypass detection. Consequently, there is a growing need to generate mutated malware by altering existing samples to comprehensively assess the robustness of ML/DL-based detectors. Unlike in the field of computer vision, functionality validation plays a crucial role in evaluating the effectiveness of these modified malware samples. Even if they achieve high evasion rates, any corruption in file format or execution can make them ineffective. To address this, we consider the essentials of functionality validation in creating malware samples by designing validators that can be used in reinforcement learning-based Windows malware mutation. Our focus is on workable and useful adversarial samples rather than the quantity. Two different functionality validation methods are proposed, leveraging the static and dynamic analysis processes of PE files to capture the representation of their behaviors to verify the preservation of designed functionalities. They are then integrated into the RL framework to support the agent in recognizing actions that can cause broken samples. Whether employing static or dynamic analysis for validation, the experimental results confirm that the proposed methods successfully maintain the original behavior of malware while enhancing its ability to evade ML-based detectors. Compared to other approaches, although the number of created adversarial malware drops due to stricter validation, a higher ratio of them are confirmed functionality-preserved. Functionality validation is an essential task in creating Windows malware mutants to ensure their reliability and usability in further assessment scenarios or real-life attacks.