[Paper Review] Defensive Quantization: When Efficiency Meets Robustness

Ji Lin, Chuang Gan+1 more|arXiv (Cornell University)|Apr 17, 2019

Adversarial Robustness in Machine Learning34 references34 citations

TL;DR

The paper shows that vanilla neural network quantization hurts adversarial robustness due to an error amplification effect, and introduces Defensive Quantization (DQ) which regularizes the Lipschitz constant during quantization to preserve efficiency while boosting robustness, demonstrated on CIFAR-10 and SVHN.

ABSTRACT

Neural network quantization is becoming an industry standard to efficiently deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and FPGAs. However, we observe that the conventional quantization approaches are vulnerable to adversarial attacks. This paper aims to raise people's awareness about the security of the quantized models, and we designed a novel quantization methodology to jointly optimize the efficiency and robustness of deep learning models. We first conduct an empirical study to show that vanilla quantization suffers more from adversarial attacks. We observe that the inferior robustness comes from the error amplification effect, where the quantization operation further enlarges the distance caused by amplified noise. Then we propose a novel Defensive Quantization (DQ) method by controlling the Lipschitz constant of the network during quantization, such that the magnitude of the adversarial noise remains non-expansive during inference. Extensive experiments on CIFAR-10 and SVHN datasets demonstrate that our new quantization method can defend neural networks against adversarial examples, and even achieves superior robustness than their full-precision counterparts while maintaining the same hardware efficiency as vanilla quantization approaches. As a by-product, DQ can also improve the accuracy of quantized models without adversarial attack.

Motivation & Objective

• Motivate security concerns of deployed quantized models under adversarial attacks.
• Explain why conventional quantization leads to vulnerability via error amplification.
• Propose a quantization method (DQ) that jointly enhances efficiency and robustness.
• Show empirical evidence that DQ outperforms vanilla quantization and matches or exceeds full-precision robustness on selected datasets.
• Highlight the compatibility of DQ with other defense techniques and its training benefits.

Proposed method

• Empirical study showing quantization suffers from adversarial attacks due to error amplification across layers.
• Introduce Defensive Quantization (DQ) by constraining the Lipschitz constant of the network to prevent perturbation amplification.
• Regularize weights with a term ||W^T W − I||^2 to keep layer Lipschitz constants ≤ 1 (non-expansive).
• Apply activation quantization (ReLU6) and integrate Lipschitz regularization into the training objective: L = L_CE + (β/2) sum_W ||W^T W − I||^2.
• Use convex aggregation adjustments for ResNets as in prior work to maintain stability.
• Demonstrate that DQ can be combined with other defenses (e.g., adversarial training, feature squeezing) for enhanced robustness.

Experimental results

Research questions

• RQ1 How does conventional quantization affect robustness against adversarial examples compared to full-precision models?
• RQ2 Can Lipschitz-constant control during quantization prevent adversarial perturbation amplification across layers?
• RQ3 Does Defensive Quantization preserve hardware efficiency while improving or maintaining robustness?
• RQ4 How well does DQ cooperate with other defense strategies to raise overall adversarial robustness?

Key findings

• Conventional activation quantization makes quantized models less robust to adversarial attacks than full-precision ones, even when clean accuracy is preserved.
• Defensive Quantization (DQ) bridges or surpasses the robustness of full-precision models while maintaining the same hardware efficiency as vanilla quantization.
• Regularizing to keep layer Lipschitz constants non-expansive reduces adversarial noise propagation, with larger β yielding greater robustness gains.
• DQ can improve the training of quantized models on clean data by constraining the activation dynamic range and reducing truncation-induced optimization difficulty.
• DQ is compatible with and can further improve the effectiveness of other defenses (e.g., adversarial training, feature squeezing).