[Paper Review] Semantic-preserving Reinforcement Learning Attack Against Graph Neural Networks for Malware Detection

Lan Zhang, Peng Liu+2 more|arXiv (Cornell University)|Sep 11, 2020

Advanced Malware Detection Techniques53 references4 citations

TL;DR

This paper proposes a reinforcement learning-based attack that generates semantics-preserving adversarial malware by inserting semantic Nops into control flow graphs (CFGs) to evade black-box graph neural network (GNN) malware detectors. The method automatically selects optimal basic blocks and semantic Nops using an RL agent, achieving significantly higher evasion rates than baseline attacks.

ABSTRACT

As an increasing number of deep-learning-based malware scanners have been proposed, the existing evasion techniques, including code obfuscation and polymorphic malware, are found to be less effective. In this work, we propose a reinforcement learning-based semantics-preserving (i.e.functionality-preserving) attack against black-box GNNs (GraphNeural Networks) for malware detection. The key factor of adversarial malware generation via semantic Nops insertion is to select the appropriate semanticNopsand their corresponding basic blocks. The proposed attack uses reinforcement learning to automatically make these "how to select" decisions. To evaluate the attack, we have trained two kinds of GNNs with five types(i.e., Backdoor, Trojan-Downloader, Trojan-Ransom, Adware, and Worm) of Windows malware samples and various benign Windows programs. The evaluation results have shown that the proposed attack can achieve a significantly higher evasion rate than three baseline attacks, namely the semantics-preserving random instruction insertion attack, the semantics-preserving accumulative instruction insertion attack, and the semantics-preserving gradient-based instruction insertion attack.

Motivation & Objective

• To address the growing effectiveness of GNN-based malware detection models that render traditional evasion techniques like code obfuscation less effective.
• To develop a semantics-preserving attack that maintains malware functionality while evading GNN-based detectors.
• To automate the decision-making process of selecting which basic blocks and semantic Nops to insert using reinforcement learning.
• To evaluate the attack's effectiveness against black-box GNN models trained on Windows malware and benign programs.
• To explore the feasibility of integrating adversarial example generation with existing code obfuscation techniques for reduced implementation overhead.

Proposed method

• The attack uses a reinforcement learning agent to decide where and which semantic Nops to insert into control flow graphs (CFGs) of malware samples.
• The RL agent is trained using a reward function that encourages evasion of GNN-based malware detectors while preserving program semantics.
• Semantic Nops are inserted into basic blocks of the CFG to alter the graph structure without changing the program's functionality.
• The attack operates in a black-box setting, requiring no access to the internal weights or gradients of the target GNN model.
• Two types of GNN models (GCN and GAT) are trained on CFGs derived from Windows malware and benign programs for evaluation.
• The method is evaluated against four baselines: random insertion, accumulative insertion, gradient-based insertion, and binary diversification.

Experimental results

Research questions

• RQ1 Can reinforcement learning be effectively used to automate the selection of semantic Nops and their insertion locations in CFGs to evade GNN-based malware detectors?
• RQ2 How does the proposed semantics-preserving attack compare in evasion rate to existing baseline methods such as random or gradient-based instruction insertion?
• RQ3 To what extent does the attack maintain malware functionality while evading detection?
• RQ4 Can the attack be applied to other GNN-based models or sequential data representations like instruction or API call sequences?

Key findings

• The proposed reinforcement learning-based attack achieved a significantly higher evasion rate than all four baseline attacks on both GCN and GAT models.
• The attack successfully evaded GNN-based malware detectors without altering the functional behavior of the malware samples.
• The RL agent automatically learned optimal insertion strategies without requiring gradient information or white-box access to the target model.
• The method outperformed the semantics-preserving random instruction insertion attack, the accumulative instruction insertion attack, and the gradient-based instruction insertion attack in terms of evasion success rate.
• The results demonstrate that GNN-based malware detectors are vulnerable to semantics-preserving attacks when adversarial modifications are made at the basic block level via strategic Nop insertion.