[Paper Review] Is AmI (Attacks Meet Interpretability) Robust to Adversarial Examples?

Nicholas Carlini|arXiv (Cornell University)|Feb 6, 2019

Adversarial Robustness in Machine Learning3 references18 citations

TL;DR

This paper evaluates the AmI defense, which uses interpretability to identify important neurons and reject adversarial inputs by comparing predictions from the original and a modified network. Despite being presented as robust, the authors demonstrate that a simple, defense-oblivious attack achieves 100% success in evading detection on untargeted adversarial examples with an ℓ∞ bound of 0.01, reducing the detection rate to 0%, proving the defense ineffective.

ABSTRACT

No.

Motivation & Objective

• To evaluate the robustness of the AmI defense, which uses interpretability to detect adversarial examples in face recognition.
• To investigate whether the defense remains effective under a weak threat model where the attacker is oblivious to its existence.
• To challenge the validity of defense claims when attacks fail, emphasizing the need for rigorous evaluation.
• To demonstrate that even simple, high-confidence targeted attacks on the original model can evade detection by AmI.
• To encourage transparency in defense research by advocating for source code release and critical evaluation of attack efficacy.

Proposed method

• The authors generate adversarial examples using only the original, unmodified neural network, ignoring the AmI defense entirely.
• They select a random incorrect target label and generate a high-confidence targeted adversarial example on the original model.
• The adversarial example is tested on both the original and the augmented AmI network; if both classify it the same way, it is considered to evade detection.
• The process is repeated until a successful evasion is found, with a median of 25 attempts required.
• The attack is defense-oblivious, meaning it does not adapt to the structure or behavior of the AmI defense.
• The method relies on the assumption that if the original and augmented networks agree on the prediction, the input is not rejected by AmI.

Experimental results

Research questions

• RQ1 Can a defense-oblivious attack successfully evade the AmI detection mechanism?
• RQ2 Is the AmI defense robust to untargeted adversarial examples with an ℓ∞ bound of 0.01?
• RQ3 Why do attacks fail to detect adversarial examples in AmI, and what does this imply about the defense's validity?
• RQ4 Does the use of interpretability techniques in AmI genuinely improve robustness against adversarial inputs?
• RQ5 To what extent does the lack of a clear threat model in the original defense paper hinder proper security evaluation?

Key findings

• The AmI defense has a 0% true-positive rate on untargeted adversarial examples, meaning it fails to detect any such examples.
• The attack achieves 100% success in evading detection, with the median number of attempts required being 25.
• Even under a weak threat model where the attacker is oblivious to the defense, the defense is completely ineffective.
• The detection rate is lower than the 9.9% false positive rate of the original undefended model, indicating worse performance.
• The authors confirm that the defense's claimed robustness is invalid, as it fails under minimal assumptions.
• The study highlights the risk of self-deception in evaluating defenses when attacks fail without clear reasons.