[Paper Review] Robust Physical-World Attacks on Machine Learning Models.

Ivan Evtimov, Kevin Eykholt+6 more|arXiv (Cornell University)|Jul 27, 2017

Adversarial Robustness in Machine LearningComputer Science10 references197 citations

TL;DR

This paper introduces Robust Physical Perturbations (RP2), a novel attack algorithm that generates spatially constrained adversarial perturbations mimicking vandalism or art to evade detection in the physical world. RP2 achieves 100% success in fooling real road sign recognition systems under diverse physical conditions, including a Stop sign misclassified as a Speed Limit sign and a Right Turn sign misclassified as either a Stop or Added Lane sign.

ABSTRACT

Deep neural network-based classifiers are known to be vulnerable to adversarial examples that can fool them into misclassifying their input through the addition of small-magnitude perturbations. However, recent studies have demonstrated that such adversarial examples are not very effective in the physical world--they either completely fail to cause misclassification or only work in restricted cases where a relatively complex image is perturbed and printed on paper. In this paper we propose a new attack algorithm--Robust Physical Perturbations (RP2)-- that generates perturbations by taking images under different conditions into account. Our algorithm can create spatially-constrained perturbations that mimic vandalism or art to reduce the likelihood of detection by a casual observer. We show that adversarial examples generated by RP2 achieve high success rates under various conditions for real road sign recognition by using an evaluation methodology that captures physical world conditions. We physically realized and evaluated two attacks, one that causes a Stop sign to be misclassified as a Speed Limit sign in 100% of the testing conditions, and one that causes a Right Turn sign to be misclassified as either a Stop or Added Lane sign in 100% of the testing conditions.

Motivation & Objective

• To address the limited effectiveness of existing adversarial examples in real-world physical settings, where perturbations often fail due to environmental variability.
• To develop a method that generates adversarial perturbations resilient to real-world image transformations such as viewpoint changes, lighting, and printing artifacts.
• To create perturbations that are visually inconspicuous by mimicking vandalism or artistic modifications, reducing observer detection.
• To evaluate the robustness of adversarial examples under realistic physical-world conditions, including multiple camera angles and environmental factors.
• To demonstrate the feasibility of reliable physical-world attacks on deployed machine learning models, particularly in safety-critical applications like road sign recognition.

Proposed method

• RP2 formulates the adversarial attack as an optimization problem that incorporates multiple physical-world image transformations, such as rotation, scaling, and blur, during perturbation generation.
• The method uses a differentiable rendering pipeline to simulate realistic image distortions under various camera viewpoints and lighting conditions.
• It enforces spatial constraints on perturbations to localize changes in a way that resembles real-world vandalism or artistic alterations, enhancing stealthiness.
• The algorithm optimizes for high fooling rates across a diverse set of physical conditions, ensuring robustness to real-world variability.
• It employs a multi-view evaluation strategy that simulates real-world deployment by testing under multiple camera angles and environmental settings.
• The approach integrates adversarial training-like regularization to improve generalization across different physical perturbations and transformations.

Experimental results

Research questions

• RQ1 Can adversarial perturbations be made robust to real-world physical transformations such as viewpoint changes, lighting variations, and printing artifacts?
• RQ2 How effective are adversarial examples that mimic vandalism or artistic modifications in evading human detection while maintaining high fooling rates?
• RQ3 Can a single adversarial perturbation achieve consistent misclassification across multiple physical conditions and camera angles for real-world road signs?
• RQ4 What is the success rate of physical-world attacks on deployed machine learning models when evaluated under realistic, diverse environmental conditions?
• RQ5 To what extent can adversarial examples be made both stealthy and effective in real-world deployment scenarios?

Key findings

• The RP2 attack achieved a 100% success rate in misclassifying a Stop sign as a Speed Limit sign across all tested physical conditions.
• The attack on a Right Turn sign successfully caused misclassification as either a Stop or Added Lane sign in 100% of the testing conditions.
• The perturbations generated by RP2 were visually indistinguishable from natural vandalism or artistic modifications, significantly reducing observer detection risk.
• The method demonstrated high robustness under diverse physical transformations, including changes in viewpoint, lighting, and image blur.
• The evaluation methodology successfully captured real-world variability, confirming that RP2 outperforms prior physical-world attacks in reliability and generalization.
• Physical realization of the attacks confirmed that the adversarial examples remain effective when printed and viewed under real-world conditions.