[Paper Review] Creating Valid Adversarial Examples of Malware

Matouš Kozák, Martin Jureček+2 more|arXiv (Cornell University)|Jun 23, 2023

Advanced Malware Detection TechniquesComputer Science3 citations

TL;DR

This paper proposes a reinforcement learning-based framework to generate valid, functionality-preserving adversarial malware examples at the binary level, using a set of portable executable (PE) modifications. Using proximal policy optimization (PPO), the method achieves a 53.84% evasion rate against a gradient-boosted decision tree classifier and 11.41% against MalConv, with random application of the same modifications evading 11.65% of top antivirus engines on average.

ABSTRACT

Machine learning is becoming increasingly popular as a go-to approach for many tasks due to its world-class results. As a result, antivirus developers are incorporating machine learning models into their products. While these models improve malware detection capabilities, they also carry the disadvantage of being susceptible to adversarial attacks. Although this vulnerability has been demonstrated for many models in white-box settings, a black-box attack is more applicable in practice for the domain of malware detection. We present a generator of adversarial malware examples using reinforcement learning algorithms. The reinforcement learning agents utilize a set of functionality-preserving modifications, thus creating valid adversarial examples. Using the proximal policy optimization (PPO) algorithm, we achieved an evasion rate of 53.84% against the gradient-boosted decision tree (GBDT) model. The PPO agent previously trained against the GBDT classifier scored an evasion rate of 11.41% against the neural network-based classifier MalConv and an average evasion rate of 2.31% against top antivirus programs. Furthermore, we discovered that random application of our functionality-preserving portable executable modifications successfully evades leading antivirus engines, with an average evasion rate of 11.65%. These findings indicate that machine learning-based models used in malware detection systems are vulnerable to adversarial attacks and that better safeguards need to be taken to protect these systems.

Motivation & Objective

• To develop a black-box adversarial attack framework that generates functional malware examples evading machine learning-based malware detectors.
• To ensure the generated adversarial examples preserve the original malware’s functionality through validated binary modifications.
• To evaluate transferability of adversarial examples across different classifiers, including real-world antivirus engines.
• To compare the effectiveness of trained reinforcement learning agents versus random application of modifications in evasion performance.

Proposed method

• Designed a custom OpenAI Gym-compatible environment for training reinforcement learning agents on PE binary files.
• Implemented a set of 12 functionality-preserving PE modifications (e.g., inserting NOPs, modifying section headers) to alter binaries without breaking execution.
• Used proximal policy optimization (PPO) with hyperparameters γ=0.5 and α=0.0001 to train agents to maximize evasion while minimizing file size increase.
• Employed a behavior comparison method to validate functionality preservation by comparing system-level behavior before and after modification.
• Trained and evaluated agents on a dataset of 7,000 Windows malware EXEs, testing evasion against GBDT, MalConv, and top commercial AVs.
• Compared trained PPO agents with a random agent applying the same modifications and with the MAB-Malware framework as a baseline.

Experimental results

Research questions

• RQ1 Can reinforcement learning agents generate valid, functional adversarial malware examples that evade machine learning-based malware detectors in a black-box setting?
• RQ2 How effective is the PPO-based agent in evading a gradient-boosted decision tree (GBDT) classifier compared to random modification strategies?
• RQ3 What is the transferability of adversarial examples generated against a surrogate model (GBDT) to real-world antivirus engines?
• RQ4 Does random application of the same functionality-preserving modifications outperform trained RL agents in evading commercial AVs?
• RQ5 How does the proposed method compare in validity and evasion performance to existing frameworks like MAB-Malware?

Key findings

• The PPO agent achieved a 53.84% evasion rate against the GBDT classifier, with an average file size increase of 3.57%.
• The PPO agent achieved an 11.41% evasion rate against the neural network-based MalConv classifier.
• The PPO agent achieved an average evasion rate of 2.31% against top commercial antivirus engines.
• The random agent applying the same modifications achieved an average evasion rate of 11.65% against top AVs, outperforming both the PPO agent and MAB-Malware.
• The MAB-Malware framework outperformed both the PPO and random agents on GBDT (76.12% evasion) and MalConv (60.1% evasion), but had only 2.61% average evasion on AVs.
• The study found that using GBDT as a surrogate model for generating adversarial examples does not lead to effective evasion against real-world AVs, suggesting the need for better surrogate models.