[Paper Review] Generic Black-Box End-to-End Attack against RNNs and Other API Calls Based Malware Classifiers.

Ishai Rosenberg, Asaf Shabtai+2 more|arXiv (Cornell University)|Jul 19, 2017

Advanced Malware Detection Techniques28 citations

TL;DR

This paper proposes a generic black-box adversarial attack targeting RNN-based malware classifiers using API call sequences, leveraging transferability across models to evade detection without altering malware functionality. The attack is effective against diverse classifiers, including feed-forward DNNs and traditional ML models, due to a newly introduced transferability by transitivity principle.

ABSTRACT

Deep neural networks are being used to solve complex classification problems, in which other machine learning classifiers, such as SVM, fall short. Recurrent Neural Networks (RNNs) have been used for tasks that involves sequential inputs, like speech to text. In the cyber security domain, RNNs based on API calls have been able to classify unsigned malware better than other classifiers. In this paper we present a black-box attack against RNNs, focusing on finding adversarial API call sequences that would be misclassified by a RNN without affecting the malware functionality. We also show that the this attack is effective against many classifiers, due-to the transferability principle between RNN variants, feed-forward DNNs and state-of-the-art traditional machine learning classifiers. Finally, we introduce the transferability by transitivity principle, causing an attack against generalized classifier like RNN variants to be transferable to less generalized classifiers like feed-forward DNNs. We conclude by discussing possible defense mechanisms.

Motivation & Objective

• To develop a black-box adversarial attack that generates adversarial API call sequences to mislead RNN-based malware classifiers.
• To investigate the transferability of adversarial examples across different classifier architectures, including RNN variants, feed-forward DNNs, and traditional ML models.
• To establish a theoretical foundation for transferability by transitivity, enabling attacks on generalized models to generalize to less generalized ones.
• To evaluate the effectiveness of the attack in maintaining malware functionality while evading detection.
• To explore potential defense mechanisms against such transferable adversarial attacks in malware classification systems.

Proposed method

• The attack generates adversarial API call sequences by perturbing the input sequence in a way that misleads the RNN classifier while preserving the original malware's functionality.
• The method relies on the transferability principle, where adversarial examples crafted for one model (e.g., RNN) successfully fool other models (e.g., feed-forward DNNs, SVMs).
• A new principle—transferability by transitivity—is introduced, enabling attacks on generalized models (like RNNs) to be effective against less generalized models (like feed-forward DNNs).
• The attack operates entirely in the black-box setting, requiring no access to model gradients or internal parameters.
• The approach uses iterative optimization to generate adversarial sequences that maximize misclassification while minimizing detectable changes to the API call flow.
• The method is evaluated across multiple classifier types, demonstrating consistent success across RNNs, feed-forward DNNs, and traditional ML models.

Experimental results

Research questions

• RQ1 Can adversarial API call sequences be generated in a black-box setting that evade RNN-based malware classifiers without altering malware functionality?
• RQ2 To what extent do adversarial examples transfer across different types of classifiers, including RNNs, feed-forward DNNs, and traditional ML models?
• RQ3 Does the transferability of adversarial examples extend from generalized models (e.g., RNNs) to less generalized models (e.g., feed-forward DNNs) via the principle of transitivity?
• RQ4 How effective is the proposed attack in maintaining malware functionality while achieving high evasion rates?
• RQ5 What are the implications of this attack for the robustness of current malware classification systems?

Key findings

• The proposed black-box attack successfully generates adversarial API call sequences that evade RNN-based malware classifiers with high success rates.
• Adversarial examples crafted for RNNs transfer effectively to feed-forward DNNs and traditional machine learning classifiers like SVMs, demonstrating strong cross-model transferability.
• The introduction of the transferability by transitivity principle enables attacks on generalized models to be effective against less generalized models, expanding the attack surface.
• The attack maintains the original malware functionality, ensuring that the malicious behavior remains intact despite the adversarial perturbations.
• The method is effective across diverse classifier architectures, indicating a systemic vulnerability in current API-based malware classification systems.
• The results suggest that current defenses based on model-specific robustness may be insufficient against transferable adversarial attacks in the black-box setting.