[Paper Review] SentiNet: Detecting Physical Attacks Against Deep Learning Systems

Edward Chou, Florian Tramèr+2 more|arXiv (Cornell University)|Dec 4, 2018

Adversarial Robustness in Machine LearningComputer Science34 references84 citations

TL;DR

SentiNet is a novel, attack-agnostic detection framework that leverages model interpretability and object detection to identify localized universal adversarial attacks—such as physical patches and data poisoning—without requiring prior knowledge of the attack or retraining. It achieves strong performance across diverse attack types and remains robust against adaptive adversaries designing patches to evade detection.

ABSTRACT

SentiNet is a novel detection framework for localized universal attacks on neural networks. These attacks restrict adversarial noise to contiguous portions of an image and are reusable with different images -- constraints that prove useful for generating physically-realizable attacks. Unlike most other works on adversarial detection, SentiNet does not require training a model or preknowledge of an attack prior to detection. Our approach is appealing due to the large number of possible mechanisms and attack-vectors that an attack-specific defense would have to consider. By leveraging the neural network's susceptibility to attacks and by using techniques from model interpretability and object detection as detection mechanisms, SentiNet turns a weakness of a model into a strength. We demonstrate the effectiveness of SentiNet on three different attacks -- i.e., data poisoning attacks, trojaned networks, and adversarial patches (including physically realizable attacks) -- and show that our defense is able to achieve very competitive performance metrics for all three threats. Finally, we show that SentiNet is robust against strong adaptive adversaries, who build adversarial patches that specifically target the components of SentiNet's architecture.

Motivation & Objective

• To address the challenge of detecting physically realizable, localized universal adversarial attacks on deep neural networks.
• To develop a defense mechanism that does not require prior knowledge of the attack or model retraining.
• To create a detection framework resilient against adaptive adversaries who craft attacks specifically to evade detection.
• To generalize across diverse attack types, including adversarial patches, data poisoning, and trojaned models.

Proposed method

• SentiNet uses class activation mapping (CAM) to identify salient regions in input images that are most influential for model predictions.
• It applies object detection techniques to localize suspicious, high-activation regions that may correspond to adversarial perturbations.
• The framework treats the neural network's attention mechanisms as indicators of potential attacks, transforming model susceptibility into a detection signal.
• It combines interpretability maps with object detection to detect localized, reusable adversarial noise across different inputs.
• The system is designed to be modular and attack-agnostic, avoiding reliance on specific attack patterns or training data.
• It is evaluated against adaptive adversaries who optimize patches to evade SentiNet’s detection components.

Experimental results

Research questions

• RQ1 Can a detection framework identify localized universal adversarial attacks without prior knowledge of the attack or model retraining?
• RQ2 How effective is SentiNet in detecting physically realizable adversarial patches across different models and datasets?
• RQ3 To what extent is SentiNet robust against adaptive adversaries who design attacks to evade its detection mechanisms?
• RQ4 Can interpretability-based detection be generalized across diverse attack types such as data poisoning and model trojaning?
• RQ5 How does SentiNet’s performance compare to attack-specific defenses in terms of detection accuracy and robustness?

Key findings

• SentiNet achieves competitive detection performance across three distinct attack types: adversarial patches, data poisoning, and trojaned models.
• The framework successfully detects physically realizable adversarial patches even when they are optimized to evade detection.
• SentiNet remains robust against strong adaptive adversaries who craft patches specifically to bypass its detection components.
• The method does not require retraining or prior knowledge of the attack, making it broadly applicable and practical for real-world deployment.
• By leveraging model interpretability and object detection, SentiNet transforms a model’s vulnerability into a detection advantage.
• The approach demonstrates high detection accuracy without relying on attack-specific signatures or training data.