[Paper Review] Physical Adversarial Textures that Fool Visual Object Tracking

Rey Reza Wiyatno, Anqi Xu|arXiv (Cornell University)|Apr 24, 2019

Adversarial Robustness in Machine Learning25 references11 citations

TL;DR

This paper introduces Physical Adversarial Textures (PATs)—inconspicuous poster-like patterns that fool visual object tracking systems in the real world. By optimizing guided adversarial losses within an Expectation Over Transformation (EOT) framework, the method generates textures that cause trackers like GOTURN to lose target, even under diverse lighting, angles, and viewing conditions, demonstrating successful sim-to-real transfer and robust adversarial behavior.

ABSTRACT

We present a system for generating inconspicuous-looking textures that, when displayed in the physical world as digital or printed posters, cause visual object tracking systems to become confused. For instance, as a target being tracked by a robot's camera moves in front of such a poster, our generated texture makes the tracker lock onto it and allows the target to evade. This work aims to fool seldom-targeted regression tasks, and in particular compares diverse optimization strategies: non-targeted, targeted, and a new family of guided adversarial losses. While we use the Expectation Over Transformation (EOT) algorithm to generate physical adversaries that fool tracking models when imaged under diverse conditions, we compare the impacts of different conditioning variables, including viewpoint, lighting, and appearances, to find practical attack setups with high resulting adversarial strength and convergence speed. We further showcase textures optimized solely using simulated scenes can confuse real-world tracking systems.

Motivation & Objective

• To develop physically realizable adversarial textures that disrupt visual object tracking systems in real-world settings.
• To evaluate and compare different adversarial loss objectives—non-targeted, targeted, and a novel guided adversarial loss—for improving attack effectiveness.
• To study the impact of scene variable randomization in EOT to identify minimal yet effective transformations for robust physical adversaries.
• To demonstrate sim-to-real transfer of PATs generated in a non-photorealistic simulator using diffuse materials.
• To highlight the vulnerability of vision-only tracking systems to subtle, inconspicuous physical patterns.

Proposed method

• The method uses an Expectation Over Transformation (EOT) framework to optimize adversarial textures under random variations in camera pose, lighting, background, and object positioning.
• A new family of guided adversarial losses is introduced, which balances between non-targeted and targeted objectives to improve convergence and adversarial strength.
• The attack optimizes textures using a differentiable rendering pipeline that simulates real-world imaging conditions, including perspective, blur, and lighting.
• The optimization process applies gradient-based updates to texture patterns while constraining perturbations to remain visually inconspicuous.
• The method evaluates the robustness of PATs across diverse physical conditions, including varying distances, angles, and motion blur.
• Sim-to-real transfer is validated by deploying textures from a non-photorealistic simulator onto physical posters, which successfully fooled real-world trackers.

Experimental results

Research questions

• RQ1 Can physically realizable adversarial textures be generated that cause visual object trackers to lose track of a target in real-world settings?
• RQ2 How do different adversarial loss objectives—non-targeted, targeted, and guided—impact the convergence speed and strength of physical adversarial attacks?
• RQ3 Which scene variables in the EOT framework must be randomized to achieve robust physical adversarial performance without unnecessary computational cost?
• RQ4 To what extent can adversarial textures trained in a non-photorealistic simulator successfully transfer to real-world tracking systems?
• RQ5 Can physically inconspicuous textures induce persistent tracker misclassification, even when the target reappears after being obscured?

Key findings

• The guided adversarial loss family significantly improved both convergence speed and adversarial strength compared to non-targeted and targeted losses.
• Physical adversarial textures containing 'striped patches' consistently caused GOTURN to lock onto the texture rather than the target, even after the target reappeared.
• The attack achieved strong performance in 57 out of 80 stationary tracking runs and 6 out of 18 servoing runs, demonstrating robustness under dynamic conditions.
• PATs optimized in a non-photorealistic simulator successfully transferred to real-world settings, fooling cameras and robotic tracking systems.
• Randomizing only key scene variables—such as camera position, lighting, and object placement—was sufficient to generate strong physical adversaries, avoiding unnecessary computational overhead.
• The success of sim-to-real transfer may be aided by GOTURN’s difficulty distinguishing synthetic patterns from real targets, suggesting a potential mechanism for real-world effectiveness.