[Paper Review] Lipschitz regularity of deep neural networks: analysis and efficient estimation

Kevin Scaman, Aladin Virmaux|arXiv (Cornell University)|May 28, 2018

Adversarial Robustness in Machine Learning24 references135 citations

TL;DR

The paper proves exact Lipschitz computation for neural nets is NP-hard, and introduces AutoLip and SeqLip to efficiently upper-bound Lipschitz constants for automatically differentiable functions, with experiments showing SeqLip substantially improves AutoLip bounds on various architectures and an accompanying PyTorch implementation.

ABSTRACT

Deep neural networks are notorious for being sensitive to small well-chosen perturbations, and estimating the regularity of such architectures is of utmost importance for safe and robust practical applications. In this paper, we investigate one of the key characteristics to assess the regularity of such methods: the Lipschitz constant of deep learning architectures. First, we show that, even for two layer neural networks, the exact computation of this quantity is NP-hard and state-of-art methods may significantly overestimate it. Then, we both extend and improve previous estimation methods by providing AutoLip, the first generic algorithm for upper bounding the Lipschitz constant of any automatically differentiable function. We provide a power method algorithm working with automatic differentiation, allowing efficient computations even on large convolutions. Second, for sequential neural networks, we propose an improved algorithm named SeqLip that takes advantage of the linear computation graph to split the computation per pair of consecutive layers. Third we propose heuristics on SeqLip in order to tackle very large networks. Our experiments show that SeqLip can significantly improve on the existing upper bounds. Finally, we provide an implementation of AutoLip in the PyTorch environment that may be used to better estimate the robustness of a given neural network to small perturbations or regularize it using more precise Lipschitz estimations.

Motivation & Objective

• Motivate and formalize Lipschitz regularity as a measure of neural network robustness.
• Prove NP-hardness of exact Lipschitz constant computation even for simple networks.
• Develop practical upper-bound estimators AutoLip and SeqLip for automatically differentiable functions.
• Provide efficient, differentiable-graph compatible implementations for computing Lipschitz bounds.
• Evaluate the methods on common architectures and provide an open-source PyTorch implementation.

Proposed method

• Define Lipschitz constant for vector-valued functions and relate it to the operator norm of the Jacobian (via Rademacher’s theorem).
• Show NP-hardness of exact Lipschitz computation for a 2-layer MLP with ReLU, motivating upper-bound methods.
• Introduce AutoLip: an algorithm that upper-bounds L(f) using automatic differentiation over the computation graph (Alg. 1).
• Extend a power-method-based approach to compute spectral norms of affine layers (linear/convolution) via autograd (Alg. 2).
• Develop SeqLip: an improved bound for sequential networks by splitting the computation graph per layer and using SVD-based decomposition (Eq. 8).
• Propose Greedy SeqLip and low-rank approximations for large layers to manage complexity; provide theoretical intuition and bounds (Theorem 3).
• Discuss concrete Lipschitz constants for common layers (activation, pooling, normalization) and provide practical notes for implementation.

Experimental results

Research questions

• RQ1 Is the exact Lipschitz constant of a neural network computable in polynomial time?
• RQ2 How can we upper-bound the Lipschitz constant for functions that are automatically differentiable, like neural nets?
• RQ3 Can we improve existing Lipschitz upper bounds for sequential networks beyond simple per-layer products?
• RQ4 How do AutoLip and SeqLip perform on standard architectures (MLPs, CNNs, AlexNet) and can we implement them in PyTorch?
• RQ5 What theoretical guarantees explain when SeqLip provides substantial improvements over AutoLip?

Key findings

• Exact Lipschitz computation is NP-hard even for simple 2-layer MLPs with ReLU activations.
• AutoLip provides a generic upper bound for the Lipschitz constant of any automatically differentiable function.
• SeqLip yields tighter upper bounds for sequential networks by exploiting the computation graph structure and layerwise decompositions.
• SeqLip can significantly tighten AutoLip bounds in practice, sometimes by up to a factor of 8 for MLPs and CNNs.
• When used on architectures like AlexNet, AutoLip yields very large upper bounds, while SeqLip can achieve substantial reductions.
• An open-source PyTorch implementation (lipEstimation) is provided to estimate robustness of networks to small perturbations.