[Paper Review] Model Compression with Adversarial Robustness: A Unified Optimization Framework

Shupeng Gui, Haotao N. Wang+4 more|arXiv (Cornell University)|Feb 10, 2019

Adversarial Robustness in Machine Learning62 citations

TL;DR

This paper introduces ATMC, a unified constrained optimization framework that jointly performs adversarial training with pruning, low-rank factorization, and nonuniform quantization to produce compact, robust CNNs. It uses ADMM to handle the combined constraints and demonstrates favorable robustness–model size trade-offs compared to baselines.

ABSTRACT

Deep model compression has been extensively studied, and state-of-the-art methods can now achieve high compression ratios with minimal accuracy loss. This paper studies model compression through a different lens: could we compress models without hurting their robustness to adversarial attacks, in addition to maintaining accuracy? Previous literature suggested that the goals of robustness and compactness might sometimes contradict. We propose a novel Adversarially Trained Model Compression (ATMC) framework. ATMC constructs a unified constrained optimization formulation, where existing compression means (pruning, factorization, quantization) are all integrated into the constraints. An efficient algorithm is then developed. An extensive group of experiments are presented, demonstrating that ATMC obtains remarkably more favorable trade-off among model size, accuracy and robustness, over currently available alternatives in various settings. The codes are publicly available at: https://github.com/shupenggui/ATMC.

Motivation & Objective

• Motivate the need for compressing CNNs without sacrificing adversarial robustness in resource-constrained environments (e.g., IoT).
• Design a unified framework that integrates pruning, low-rank factorization, and quantization under adversarial robustness constraints.
• Develop an efficient optimization algorithm (ADMM-based) to solve the constrained min–max problem.
• Provide empirical evidence that ATMC yields better robustness–size trade-offs than sequential or naive baselines.

Proposed method

• Formulate ATMC as a constrained min–max optimization where adversarial training minimizes a worst-case loss over bounded perturbations.
• Impose a unified sparsity/structure constraint W = UV + C with total nonzeros ||U||0 + ||V||0 + ||C||0 ≤ k and a quantization constraint |θ|0 ≤ 2^b per nonzero element.
• Use nonuniform quantization learned during training by constraining the number of unique nonzero values via a bit-precision parameter b.
• Apply ADMM to split constraints and enable alternating updates for θ, θ′, and dual variables, including a ZeroKmeans-based projection for quantization.
• Represent the quantization via a clustering-like projection that assigns nonzero weights to a small set of learned values (including zero).
• Provide an algorithmic outline (ZeroKmeans for quantization, sparse projection for pruning, and adversarial example generation during training).

Experimental results

Research questions

• RQ1 Can a single optimization framework jointly achieve model compression and adversarial robustness across pruning, factorization, and quantization?
• RQ2 Do ATMC-compressed models offer superior robustness-accuracy and compression trade-offs compared with baselines (pure compression, pure defense, or mixtures)?
• RQ3 Is the robustness of ATMC-compressed models generalizable to different adversarial attackers and perturbation levels?

Key findings

• ATMC consistently achieves more favorable trade-offs between model size, accuracy, and robustness than baseline combinations.
• Joint optimization with ATMC can reach competitive benign accuracy and robustness at high compression ratios.
• Quantization (8-bit) enables more aggressive compression with preserved robustness relative to 32-bit variants.
• ATMC-compressed models exhibit robustness comparable to denser adversarially trained models while requiring far fewer parameters.
• Compression with ATMC can reduce the gap between benign accuracy and attacked robustness observed in naïve compression methods.