[Paper Review] Adversarially Optimized Mixup for Robust Classification

Jason Bunk, Srinjoy Chattopadhyay+2 more|arXiv (Cornell University)|Mar 22, 2021

Adversarial Robustness in Machine Learning27 references4 citations

TL;DR

This paper proposes Adversarially Optimized Mixup, a method that enhances adversarial robustness by jointly optimizing mixup interpolation and adversarial perturbations via backpropagation through the interpolation process. By using projected gradient descent to find worst-case points in the mixup space, the approach improves robust accuracy by up to 2.1% on CIFAR-10 and 4% on CIFAR-100 under strong adversaries like PGD-100 and AutoAttack.

ABSTRACT

Mixup is a procedure for data augmentation that trains networks to make smoothly interpolated predictions between datapoints. Adversarial training is a strong form of data augmentation that optimizes for worst-case predictions in a compact space around each data-point, resulting in neural networks that make much more robust predictions. In this paper, we bring these ideas together by adversarially probing the space between datapoints, using projected gradient descent (PGD). The fundamental approach in this work is to leverage backpropagation through the mixup interpolation during training to optimize for places where the network makes unsmooth and incongruous predictions. Additionally, we also explore several modifications and nuances, like optimization of the mixup ratio and geometrical label assignment, and discuss their impact on enhancing network robustness. Through these ideas, we have been able to train networks that robustly generalize better; experiments on CIFAR-10 and CIFAR-100 demonstrate consistent improvements in accuracy against strong adversaries, including the recent strong ensemble attack AutoAttack. Our source code would be released for reproducibility.

Motivation & Objective

• Address the overfitting problem in adversarial training, where robust models still exhibit low accuracy under strong adversarial attacks.
• Improve generalization of adversarially trained networks by leveraging insights from data augmentation in standard training.
• Overcome limitations of prior mixup-adversarial training methods that fail to fully probe the worst-case regions in the interpolation space between data points.
• Develop a method that enables end-to-end backpropagation through both mixup interpolation and adversarial optimization to learn smoother, more robust decision boundaries.
• Demonstrate robustness gains against strong, state-of-the-art adversaries, including gradient-free black-box attacks, without relying on gradient obfuscation.

Proposed method

• Integrate mixup augmentation with adversarial training by performing adversarial optimization (via PGD) directly in the interpolated space between data points.
• Backpropagate through the mixup interpolation process to jointly optimize the input perturbations and the mixing ratio $\lambda$.
• Use a sigmoid-based clipping function to constrain the mixing ratio $\lambda$ between 0.5 and 1.0, ensuring meaningful interpolation while enabling end-to-end optimization.
• Apply projected gradient descent (PGD) to find the worst-case adversarial perturbations in the interpolated space, maximizing the KL divergence between predictions and smoothed labels.
• Explore geometric label assignment and adaptive mixing ratio optimization to improve robustness, with ablation studies validating component contributions.
• Train models using a constrained min-max formulation that optimizes for the worst-case loss in the vicinal neighborhood defined by mixup.

Experimental results

Research questions

• RQ1 Can adversarial optimization in the mixup interpolation space lead to more robust generalization than standard adversarial training or mixup alone?
• RQ2 Does end-to-end backpropagation through mixup and adversarial perturbations improve robustness by identifying more severe failure points in the decision boundary?
• RQ3 How does the choice of mixing ratio and label assignment strategy affect robust accuracy under strong adversaries?
• RQ4 Can the proposed method improve robustness on challenging datasets like CIFAR-100, where data scarcity amplifies the need for effective augmentation?
• RQ5 Does the method remain robust against gradient-free black-box attacks, indicating it does not rely on gradient obfuscation?

Key findings

• The proposed method achieves a 2.1% absolute improvement in robust accuracy over the strongest baseline on CIFAR-10 under a 100-step PGD attack.
• On CIFAR-100, the method improves robust accuracy by approximately 4% under PGD-100, demonstrating greater benefit on more challenging, low-data regimes.
• The model achieves 58.34% robust accuracy on CIFAR-100 under AutoAttack, a strong ensemble adversary, outperforming all baselines including the 'Mix-then-attack' method.
• Ablation studies confirm that joint optimization of the mixing ratio and adversarial perturbations is critical, with the full method outperforming ablated variants by 0.6% under PGD-100.
• The method remains robust against gradient-free black-box attacks, indicating it does not suffer from gradient obfuscation and generalizes well to unseen attack types.
• The approach generalizes across architectures and shows promise when combined with larger, robustness-oriented models like RobNets for further gains.