[Paper Review] Invisible Mask: Practical Attacks on Face Recognition with Infrared

Zhe Zhou, Di Tang+4 more|arXiv (Cornell University)|Mar 13, 2018

Adversarial Robustness in Machine Learning26 references70 citations

TL;DR

The paper introduces an infrared-based adversarial attack that uses unseen infrared LEDs mounted on a cap to create an invisible perturbation on a face, enabling dodging surveillance and impersonation against face recognition systems such as FaceNet.

ABSTRACT

Accurate face recognition techniques make a series of critical applications possible: policemen could employ it to retrieve criminals' faces from surveillance video streams; cross boarder travelers could pass a face authentication inspection line without the involvement of officers. Nonetheless, when public security heavily relies on such intelligent systems, the designers should deliberately consider the emerging attacks aiming at misleading those systems employing face recognition. We propose a kind of brand new attack against face recognition systems, which is realized by illuminating the subject using infrared according to the adversarial examples worked out by our algorithm, thus face recognition systems can be bypassed or misled while simultaneously the infrared perturbations cannot be observed by raw eyes. Through launching this kind of attack, an attacker not only can dodge surveillance cameras. More importantly, he can impersonate his target victim and pass the face authentication system, if only the victim's photo is acquired by the attacker. Again, the attack is totally unobservable by nearby people, because not only the light is invisible, but also the device we made to launch the attack is small enough. According to our study on a large dataset, attackers have a very high success rate with a over 70\% success rate for finding such an adversarial example that can be implemented by infrared. To the best of our knowledge, our work is the first one to shed light on the severity of threat resulted from infrared adversarial examples against face recognition.

Motivation & Objective

• Motivate and quantify practical risks of infrared adversarial perturbations to face recognition systems.
• Develop an algorithm to generate IR LED-based adversarial examples under realistic hardware constraints.
• Design and calibrate a small, inconspicuous device to implement the adversarial perturbations in real-world settings.
• Evaluate the effectiveness of the attack on a real FR system (FaceNet) and on a large-scale face dataset (LFW).

Proposed method

• Model IR LED light spots as controllable perturbations with position, size, brightness, and color characteristics.
• Formulate an optimization to minimize the distance between the attacker's embedding and the victim's embedding under IR LED constraints.
• Synthesize adversarial examples by summing modeled light spots onto the attacker’s image and converting to a purple IR-visible perturbation.
• Design a physical device with three IR LEDs on a cap peak, plus calibration tools to align the perturbation with the generated adversarial example.
• Implement a calibration workflow that compares images with LEDs off and on to adjust spot centers and brightness to minimize embedding distance.
• Apply an Adam optimizer to minimize the loss J(f(I_syn), f(I_vtm)) with respect to spot parameters and overall amplification.

Experimental results

Research questions

• RQ1 Can infrared-based perturbations be crafted to reliably bypass or impersonate targets in FR systems under realistic hardware constraints?
• RQ2 How effective is an IR-based invisible mask against a mainstream FR model (FaceNet) and what is the achievable success rate for dodging and impersonation?
• RQ3 What is the role of calibration and manual fine-tuning in realizing practical adversarial examples with IR LEDs?
• RQ4 What are the practical considerations (threat model, device design, safety) for deploying such an IR-based attack in the wild?

Key findings

• Dodging attacks achieved a 100% success rate in the reported physical test.
• Impersonation success probability for finding actionable adversarial examples is over 70% on large-scale study over LFW data.
• A cap-based device with three 850 nm IR LEDs is sufficient to disrupt landmark pre-processing and enable both dodging and impersonation.
• An optimization-based pipeline can generate IR LED layouts that approximate imperceptible perturbations realizable by hardware.
• Calibration and fine-tuning significantly improve attack success, with manual adjustments sometimes yielding better results than raw optimization alone.