[Paper Review] HYDRA: Pruning Adversarially Robust Neural Networks

Vikash Sehwag, Shiqi Wang+2 more|arXiv (Cornell University)|Feb 24, 2020

Adversarial Robustness in Machine Learning56 references31 citations

TL;DR

HYDRA reframes pruning as an empirical risk minimization problem guided by robust training objectives, yielding highly compressed networks that retain both benign and adversarial robustness across multiple datasets and robust training methods.

ABSTRACT

In safety-critical but computationally resource-constrained applications, deep learning faces two key challenges: lack of robustness against adversarial attacks and large neural network size (often millions of parameters). While the research community has extensively explored the use of robust training and network pruning independently to address one of these challenges, only a few recent works have studied them jointly. However, these works inherit a heuristic pruning strategy that was developed for benign training, which performs poorly when integrated with robust training techniques, including adversarial training and verifiable robust training. To overcome this challenge, we propose to make pruning techniques aware of the robust training objective and let the training objective guide the search for which connections to prune. We realize this insight by formulating the pruning objective as an empirical risk minimization problem which is solved efficiently using SGD. We demonstrate that our approach, titled HYDRA, achieves compressed networks with state-of-the-art benign and robust accuracy, simultaneously. We demonstrate the success of our approach across CIFAR-10, SVHN, and ImageNet dataset with four robust training techniques: iterative adversarial training, randomized smoothing, MixTrain, and CROWN-IBP. We also demonstrate the existence of highly robust sub-networks within non-robust networks. Our code and compressed networks are publicly available at \url{https://github.com/inspire-group/compactness-robustness}.

Motivation & Objective

• Motivate a pruning method that preserves adversarial robustness under various robust training objectives.
• Develop a pruning framework that uses robust training losses to guide connectivity pruning.
• Enable high-rate pruning (high sparsity) without sacrificing either benign accuracy or robustness.
• Show generalization across multiple robustness paradigms (adversarial training, verifiable robustness, randomized smoothing, MixTrain, etc.).

Proposed method

• Formulate pruning as an empirical risk minimization problem with a robustness-aware loss.
• Use an importance-score based optimization to search for which connections to prune.
• Introduce scaled initialization of importance scores proportional to pre-trained weights to improve pruning performance.
• Integrate pruning with several robust training objectives (adversarial training, verifiable robustness via MixTrain and CROWN-IBP, randomized smoothing).
• Fine-tune only the non-pruned connections after pruning to recover performance.

Experimental results

Research questions

• RQ1 Can pruning be made aware of the robust training objective rather than relying on benign-training heuristics?
• RQ2 Does robustness-aware pruning enable high compression without sacrificing both benign and robust accuracy across diverse robustness objectives?
• RQ3 Are robust sub-networks discoverable within non-robust networks, and can pruning reveal them under various robustness goals?
• RQ4 How does HYDRA compare with existing robust-pruning approaches (Adv-LWM, Adv-ADMM) across datasets and architectures?
• RQ5 Is the approach scalable to large datasets (e.g., ImageNet) and different robustness formulations (adversarial training, verifiable robustness, smoothing, MixTrain, etc.)?
• RQ6

Key findings

• HYDRA consistently improves both benign accuracy and robustness compared to prior pruning methods across CIFAR-10, SVHN, and ImageNet.
• The scaled initialization of importance scores is a key driver of performance, enabling faster convergence and better pruning results.
• HYDRA outperforms Adv-LWM and Adv-ADMM across multiple architectures and pruning ratios, including very high pruning (up to 99%).
• The method generalizes to four robust training objectives: iterative adversarial training, randomized smoothing, MixTrain, and CROWN-IBP, with robust gains observed in era and vra metrics.
• There exist highly robust sub-networks within non-robust networks, indicating robustness potential even without full robustness during pre-training.
• Pruned networks can maintain or even exceed pre-trained robustness under stronger attacks and can be quantized further (8-bit) with minimal loss.