[Paper Review] When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks

Octavian Suciu, Radu Mărginean+3 more|arXiv (Cornell University)|Jan 1, 2018

Adversarial Robustness in Machine Learning41 references109 citations

TL;DR

This paper introduces the FAIL attacker model, a systematic framework to evaluate machine learning attacks under realistic constraints of limited knowledge and control. It demonstrates generalized transferability of evasion and poisoning attacks across diverse models and defenses, introducing StingRay—a practical, targeted poisoning attack effective against four ML applications using three algorithms and bypassing two defenses.

ABSTRACT

Attacks against machine learning systems represent a growing threat as highlighted by the abundance of attacks proposed lately. However, attacks often make unrealistic assumptions about the knowledge and capabilities of adversaries. To evaluate this threat systematically, we propose the FAIL attacker model, which describes the adversary's knowledge and control along four dimensions. The FAIL model allows us to consider a wide range of weaker adversaries that have limited control and incomplete knowledge of the features, learning algorithms and training instances utilized. Within this framework, we evaluate the generalized transferability of a known evasion attack and we design StingRay, a targeted poisoning attack that is broadly applicable---it is practical against 4 machine learning applications, which use 3 different learning algorithms, and it can bypass 2 existing defenses. Our evaluation provides deeper insights into the transferability of poison and evasion samples across models and suggests promising directions for investigating defenses against this threat.

Motivation & Objective

• To address the lack of systematic evaluation of machine learning attacks under realistic adversary assumptions.
• To model adversaries with limited knowledge and control over features, learning algorithms, and training instances.
• To evaluate the transferability of evasion and poisoning attacks across different models and learning algorithms.
• To design a broadly applicable, practical poisoning attack that bypasses existing defenses.
• To provide insights into effective defenses against transferable poisoning and evasion attacks.

Proposed method

• Propose the FAIL attacker model, defining adversary capabilities along four dimensions: knowledge of features, learning algorithms, training instances, and control over model updates.
• Evaluate generalized transferability of a known evasion attack across multiple models and datasets under the FAIL framework.
• Design StingRay, a targeted poisoning attack that manipulates model behavior by injecting carefully crafted training samples.
• Apply StingRay to four machine learning applications using three different learning algorithms (e.g., SVM, neural networks, decision trees).
• Test StingRay’s effectiveness against two existing defense mechanisms, demonstrating bypass capability.
• Use the FAIL model to systematically analyze the robustness of attacks under varying levels of adversary knowledge and control.

Experimental results

Research questions

• RQ1 How does the transferability of evasion attacks vary under realistic adversary constraints defined by the FAIL model?
• RQ2 Can a single poisoning attack be generalized across diverse machine learning applications and algorithms?
• RQ3 To what extent can existing defenses protect against transferable poisoning attacks under limited adversary knowledge?
• RQ4 What are the key factors enabling generalized transferability in poisoning attacks?
• RQ5 How does the FAIL model improve the systematic evaluation of adversarial threats in machine learning?

Key findings

• The FAIL model enables systematic evaluation of attacks under realistic adversary assumptions, revealing that weaker adversaries can still mount effective attacks.
• Evasion attacks show significant transferability across models even when the adversary has limited knowledge of the target model.
• StingRay successfully performs targeted poisoning across four different machine learning applications using three distinct learning algorithms.
• StingRay bypasses two existing defense mechanisms, demonstrating its practicality and robustness against known defenses.
• The study reveals that transferability in poisoning attacks is feasible even under constrained adversary knowledge, suggesting a broader threat surface.
• The results suggest that defenses must account for generalized transferability to be effective against real-world adversaries.