[Paper Review] Composite Adversarial Attacks

Xiaofeng Mao, Yuefeng Chen+4 more|arXiv (Cornell University)|Dec 10, 2020

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes Composite Adversarial Attacks (CAA), an automated method that searches for optimal sequences of base adversarial attack algorithms and their hyper-parameters using a multi-objective NSGA-II genetic algorithm. CAA achieves state-of-the-art attack success rates on 11 diverse defenses across $l_{ackslash infty}$, $l_2$, and unrestricted settings, outperforming 10 top-tier attacks while being 6× faster than AutoAttack.

ABSTRACT

Adversarial attack is a technique for deceiving Machine Learning (ML) models, which provides a way to evaluate the adversarial robustness. In practice, attack algorithms are artificially selected and tuned by human experts to break a ML system. However, manual selection of attackers tends to be sub-optimal, leading to a mistakenly assessment of model security. In this paper, a new procedure called Composite Adversarial Attack (CAA) is proposed for automatically searching the best combination of attack algorithms and their hyper-parameters from a candidate pool of extbf{32 base attackers}. We design a search space where attack policy is represented as an attacking sequence, i.e., the output of the previous attacker is used as the initialization input for successors. Multi-objective NSGA-II genetic algorithm is adopted for finding the strongest attack policy with minimum complexity. The experimental result shows CAA beats 10 top attackers on 11 diverse defenses with less elapsed time ( extbf{6 $ imes$ faster than AutoAttack}), and achieves the new state-of-the-art on $l_{\infty}$, $l_{2}$ and unrestricted adversarial attacks.

Motivation & Objective

• To address the sub-optimal and manual nature of current adversarial attack selection, which can lead to false assessments of model robustness.
• To automate the search for the strongest combination of attack algorithms and hyper-parameters to improve attack effectiveness.
• To develop a generalized attack framework that can represent single attacks, ensembles, and composite policies through sequential attack composition.
• To reduce attack time while maximizing success rate across diverse defenses, including those using $l_{\infty}$, $l_2$, and unrestricted perturbations.
• To evaluate the transferability of learned attack policies across models and defenses, ensuring generalizability beyond specific architectures.

Proposed method

• CAA represents an attack policy as a sequence of base attackers, where the output of one attacker serves as the input to the next, enabling progressive refinement of adversarial examples.
• The search space includes permutations of 32 base attackers, each with tunable hyper-parameters: perturbation magnitude $\epsilon$ and number of iterations $t$.
• A multi-objective NSGA-II genetic algorithm is used to optimize for two goals: maximum attack success rate and minimum computational complexity (gradient evaluations).
• The method supports both direct attack policy search (CAA_dic) on a target model and transferable policy learning (CAA_sub) for cross-model generalization.
• Identity attack (no perturbation) is included as a special case, allowing CAA to represent single attacks and ensembles as special cases of composite policies.
• The approach enables progressive attack improvement by using earlier attackers to generate examples near the decision boundary, making subsequent attacks more effective.

Experimental results

Research questions

• RQ1 Can an automated search over attack sequences and hyper-parameters outperform manually selected or tuned attacks in terms of success rate and efficiency?
• RQ2 Does composing attacks in sequence—using prior outputs as inputs—yield stronger adversarial examples than ensembling or single attacks?
• RQ3 Can attack policies learned on one model or dataset be effectively transferred to other models and defenses?
• RQ4 How does the length of the attack sequence (i.e., number of composed attackers) affect overall attack performance?
• RQ5 Which optimization strategy—random search, Bayesian optimization, reinforcement learning, or NSGA-II—yields the best trade-off between attack success and search cost?

Key findings

• CAA achieves the new state-of-the-art on $l_{\infty}$, $l_2$, and unrestricted adversarial attacks across 11 diverse defenses, surpassing 10 top-tier attack methods.
• CAA is 6× faster than AutoAttack, achieving higher attack success rates with significantly reduced computational cost.
• The attack success rate increases with policy length, especially in the unrestricted setting, where performance drops sharply beyond length 3, indicating high sensitivity to sequence composition.
• NSGA-II outperforms random search, Bayesian optimization, and reinforcement learning in both attack success and search efficiency, achieving 49.18% robust accuracy with only 3 GPU-days of search time.
• CAA_sub policies, learned on CIFAR-10, show strong transferability to multiple model architectures (ResNet, VGG, Inception), indicating generalizability across models and tasks.
• The method effectively mitigates gradient obfuscation risks by avoiding reliance on a single attack, instead leveraging complementary mechanisms through sequential composition.