[Paper Review] Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks

Yusuke Tsuzuku, Issei Sato+1 more|arXiv (Cornell University)|Feb 12, 2018

Model Reduction and Neural Networks76 citations

TL;DR

This paper introduces Lipschitz-Margin Training (LMT) to compute tight, scalable Lipschitz-based bounds and enlarge provably invariant regions around data, enabling robust certification for large neural networks.

ABSTRACT

High sensitivity of neural networks against malicious perturbations on inputs causes security concerns. To take a steady step towards robust classifiers, we aim to create neural network models provably defended from perturbations. Prior certification work requires strong assumptions on network structures and massive computational costs, and thus the range of their applications was limited. From the relationship between the Lipschitz constants and prediction margins, we present a computationally efficient calculation technique to lower-bound the size of adversarial perturbations that can deceive networks, and that is widely applicable to various complicated networks. Moreover, we propose an efficient training procedure that robustifies networks and significantly improves the provably guarded areas around data points. In experimental evaluations, our method showed its ability to provide a non-trivial guarantee and enhance robustness for even large networks.

Motivation & Objective

• Motivate certifiable robustness against adversarial perturbations for deep networks.
• Develop a scalable method to lower-bound perturbation sizes that can deceive networks across complex architectures.
• Propose a training procedure that enlarges provably guarded areas around data points.
• Provide practical Lipschitz constant bounds and an efficient certification workflow across diverse network structures.

Proposed method

• Relate Lipschitz constants to prediction margins to bound adversarial perturbations (Prop. 1 and Prop. 2).
• Compute upper bounds on network Lipschitz constants in a unified, differentiable manner for large/complex architectures.
• Develop Lipschitz-margin training (LMT) by modifying logits with a factor proportional to estimated Lipschitz bounds to enlarge the guarded area (Eq. 4).
• Provide an efficient, differentiable procedure to estimate operator norms (spectral norms) of linear components at training time (Algorithm 2).
• Aggregate Lipschitz bounds through network components using composition, addition, and concatenation rules to obtain a global bound (Theorem 2 and Corollaries).
• Show that the training procedure improves robustness and requires costs akin to a batch-size increase.

Experimental results

Research questions

• RQ1 Can Lipschitz constants be efficiently upper-bounded for large, modern networks to certify perturbation invariance?
• RQ2 Does a training procedure that enforces margins related to Lipschitz bounds enlarge provably guarded areas without sacrificing accuracy on clean data?
• RQ3 How do the proposed bounds compare in tightness and practicality to prior certification methods across network scales?
• RQ4 Is the method scalable to large networks (e.g., wide Residual Networks) and still provide meaningful robustness certificates?

Key findings

• The authors derive tighter, unified Lipschitz bounds for major network components (convolutions, batch norm, pooling, activations) and provide a differentiable, training-time approximation.
• Lipschitz-margin training (LMT) enlarges the provably guarded area around data points by enforcing margins that exceed a function of the Lipschitz bound and perturbation radius c.
• Experiments show improved certification tightness and robustness on both small networks and large 16-layer wide residual networks on SVHN, compared to baseline regularization methods.
• LMT yields non-trivial certified perturbation regions in the l2-norm and, in many cases, larger guarded areas than prior, computationally cheap methods for comparable architectures.
• Under C&W-type attacks, LMT-trained models maintain higher accuracy under perturbations than weight decay, Parseval networks, or spectral-norm regularization, with robustness gains especially notable at larger perturbation radii.