[Paper Review] TROJANZOO: Everything you ever wanted to know about neural backdoors (but were afraid to ask).

Ren Pang, Zheng Zhang+5 more|arXiv (Cornell University)|Dec 16, 2020

Adversarial Robustness in Machine Learning68 references19 citations

TL;DR

TROJANZOO is an open-source platform that systematically evaluates neural backdoor attacks and defenses in a unified framework, incorporating 12 attacks, 15 defenses, and multiple metrics. It reveals key insights such as the effectiveness of one-pixel triggers, the benefits of joint trigger and model optimization, and the vulnerability of sanitization-based defenses to adaptive attacks.

ABSTRACT

Neural backdoors represent one primary threat to the security of deep learning systems. The intensive research on this subject has produced a plethora of attacks/defenses, resulting in a constant arms race. However, due to the lack of evaluation benchmarks, many critical questions remain largely unexplored: (i) How effective, evasive, or transferable are different attacks? (ii) How robust, utility-preserving, or generic are different defenses? (iii) How do various factors (e.g., model architectures) impact their performance? (iv) What are the best practices (e.g., optimization strategies) to operate such attacks/defenses? (v) How can the existing attacks/defenses be further improved? To bridge the gap, we design and implement TROJANZOO, the first open-source platform for evaluating neural backdoor attacks/defenses in a unified, holistic, and practical manner. Thus, it has incorporated 12 representative attacks, 15 state-of-the-art defenses, 6 attack performance metrics, 10 defense utility metrics, as well as rich tools for in-depth analysis of attack-defense interactions. Leveraging TROJANZOO, we conduct a systematic study of existing attacks/defenses, leading to a number of interesting findings: (i) different attacks manifest various trade-offs among multiple desiderata (e.g., effectiveness, evasiveness, and transferability); (ii) one-pixel triggers often suffice; (iii) optimizing trigger patterns and trojan models jointly improves both attack effectiveness and evasiveness; (iv) sanitizing trojan models often introduces new vulnerabilities; (v) most defenses are ineffective against adaptive attacks, but integrating complementary ones significantly enhances defense robustness. We envision that such findings will help users select the right defense solutions and facilitate future research on neural backdoors.

Motivation & Objective

• Address the lack of standardized evaluation benchmarks for neural backdoor attacks and defenses.
• Systematically assess the trade-offs among effectiveness, evasiveness, transferability, and utility in backdoor attacks.
• Evaluate the robustness and practicality of state-of-the-art defenses against adaptive attacks.
• Identify optimal optimization strategies and architectural factors influencing attack and defense performance.
• Provide actionable insights for selecting effective defenses and guiding future research on neural backdoors.

Proposed method

• Design and implement TROJANZOO as a unified open-source platform integrating 12 representative backdoor attacks and 15 state-of-the-art defenses.
• Define and implement 6 attack performance metrics and 10 defense utility metrics for standardized evaluation.
• Support in-depth analysis of attack-defense interactions through rich diagnostic and visualization tools.
• Conduct large-scale empirical evaluations across diverse model architectures and datasets to assess performance trade-offs.
• Use joint optimization of trigger patterns and trojan models to improve attack effectiveness and evasiveness.
• Evaluate defense robustness by testing against adaptive attacks and analyzing unintended vulnerabilities introduced by sanitization.

Experimental results

Research questions

• RQ1 How do different backdoor attacks trade off between effectiveness, evasiveness, and transferability?
• RQ2 How robust and utility-preserving are existing defenses when subjected to adaptive attacks?
• RQ3 To what extent do architectural choices and optimization strategies influence attack and defense performance?
• RQ4 Can joint optimization of trigger patterns and model parameters improve attack outcomes?
• RQ5 What unintended vulnerabilities are introduced by defense mechanisms such as model sanitization?

Key findings

• One-pixel triggers are often sufficient to achieve high attack success rates, indicating minimal trigger complexity can still be highly effective.
• Joint optimization of trigger patterns and trojan models significantly enhances both attack effectiveness and evasiveness compared to isolated optimization.
• Sanitization-based defenses frequently introduce new, previously unknown vulnerabilities, undermining their reliability.
• Most existing defenses fail against adaptive attacks, but combining complementary defense strategies substantially improves overall robustness.
• Different attacks exhibit distinct trade-offs across desiderata such as effectiveness, evasiveness, and transferability, highlighting the need for tailored defense approaches.
• The platform reveals that transferability of backdoors varies significantly across attack types, with some methods showing broader transferability than others.