[论文解读] Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning

Hyrum S. Anderson, Anant Kharkar+3 more|arXiv (Cornell University)|Jan 26, 2018

Advanced Malware Detection Techniques参考文献 24被引用 181

一句话总结

作者提出一个黑箱强化学习框架，学习对 PE 文件进行变异的序列以规避静态 ML 恶意软件检测器并生成可运行的规避型恶意软件样本，发布用于研究的 OpenAI Gym 环境。

ABSTRACT

Machine learning is a popular approach to signatureless malware detection because it can generalize to never-before-seen malware families and polymorphic strains. This has resulted in its practical use for either primary detection engines or for supplementary heuristic detection by anti-malware vendors. Recent work in adversarial machine learning has shown that deep learning models are susceptible to gradient-based attacks, whereas non-differentiable models that report a score can be attacked by genetic algorithms that aim to systematically reduce the score. We propose a more general framework based on reinforcement learning (RL) for attacking static portable executable (PE) anti-malware engines. The general framework does not require a differentiable model nor does it require the engine to produce a score. Instead, an RL agent is equipped with a set of functionality-preserving operations that it may perform on the PE file. Through a series of games played against the anti-malware engine, it learns which sequences of operations are likely to result in evading the detector for any given malware sample. This enables completely black-box attacks against static PE anti-malware, and produces functional evasive malware samples as a direct result. We show in experiments that our method can attack a gradient-boosted machine learning model with evasion rates that are substantial and appear to be strongly dependent on the dataset. We demonstrate that attacks against this model appear to also evade components of publicly hosted antivirus engines. Adversarial training results are also presented: by retraining the model on evasive ransomware samples, a subsequent attack is 33% less effective. However, there are overfitting dangers when adversarial training, which we note. We release code to allow researchers to reproduce and improve this approach.

研究动机与目标

• Motivate and formalize a black-box evasion framework for static PE malware detectors.
• Demonstrate that an RL agent can learn mutation sequences that evade a gradient-boosted detector.
• Show that evasive samples can be used to improve model robustness via adversarial training.
• Provide an OpenAI Gym environment and open-source toolkit for researchers to study evasion.
• Highlight practical limitations and considerations for applying such evasion in real-world settings.

提出的方法

• Formulate malware evasion as a reinforcement learning problem where the agent mutates PE files without breaking format or functionality.
• Use an ACER agent to learn a policy over a set of PE-manipulation actions that preserve execution while altering features.
• Represent the environment state as a 2350-dimensional feature vector of PE-related metadata and byte statistics.
• Reward the agent with R for evasion and 0 otherwise, driving a black-box attack against the detector.
• Operate within a game-like setting where a sample may undergo up to ten mutations per round, with 50,000 total mutations per model training.
• Release an extensible OpenAI Gym environment and default ACER agent implementation to enable community research.

实验结果

研究问题

• RQ1 Can a reinforcement learning agent learn effective sequences of PE file mutations to evade static malware detectors under black-box conditions?
• RQ2 What are the dominant mutation strategies that emerge, and how do they generalize to unseen samples?
• RQ3 How effective are evasive samples for adversarial training to harden detectors against future evasion attempts?
• RQ4 What practical limitations affect the functionality and deployment of RL-based malware evasion in real-world settings?

主要发现

• The RL agent achieved notable evasion rates on holdout samples across datasets: VirusShare 24%, ransomware 12%, Virut 10%, BrowseFox 19% (versus a random policy of 23%, 9%, 9%, 18% respectively).
• The agent discovered evasive variants while training, with counts of 2085 (VirusShare), 1543 (ransomware), 619 (Virut), and 2444 (BrowseFox) evasions under a 50K mutation budget.
• Adversarial training with evasive ransomware samples reduced evasion effectiveness against a new ransomware attack from 12% to 8%.
• The approach can generate functioning PE malware variants, enabling usable evasive samples for model hardening and research.
• An OpenAI Gym environment (gym-malware) is released to study evasion rates against user models, samples, and RL agents.