[Paper Review] Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks

Mahyar Fazlyab, Alexander Robey+3 more|arXiv (Cornell University)|Jun 11, 2019

Adversarial Robustness in Machine LearningComputer Science108 citations

TL;DR

Presents LipSDP, a convex semidefinite-programming framework that yields tight global Lipschitz bounds for deep neural networks by modeling activations as quadratic constraints, with scalable variants and distributed implementations.

ABSTRACT

Tight estimation of the Lipschitz constant for deep neural networks (DNNs) is useful in many applications ranging from robustness certification of classifiers to stability analysis of closed-loop systems with reinforcement learning controllers. Existing methods in the literature for estimating the Lipschitz constant suffer from either lack of accuracy or poor scalability. In this paper, we present a convex optimization framework to compute guaranteed upper bounds on the Lipschitz constant of DNNs both accurately and efficiently. Our main idea is to interpret activation functions as gradients of convex potential functions. Hence, they satisfy certain properties that can be described by quadratic constraints. This particular description allows us to pose the Lipschitz constant estimation problem as a semidefinite program (SDP). The resulting SDP can be adapted to increase either the estimation accuracy (by capturing the interaction between activation functions of different layers) or scalability (by decomposition and parallel implementation). We illustrate the utility of our approach with a variety of experiments on randomly generated networks and on classifiers trained on the MNIST and Iris datasets. In particular, we experimentally demonstrate that our Lipschitz bounds are the most accurate compared to those in the literature. We also study the impact of adversarial training methods on the Lipschitz bounds of the resulting classifiers and show that our bounds can be used to efficiently provide robustness guarantees.

Motivation & Objective

• Motivate tight and scalable estimation of global Lipschitz constants for deep neural networks.
• Develop a convex optimization framework that provides guaranteed upper bounds on Lipschitz constants.
• Capture activation function properties via quadratic constraints to enable SDP formulations.
• Offer variants that trade off estimation accuracy against computational scalability.
• Demonstrate applicability to robustness certification and stability analysis in learning-augmented systems.

Proposed method

• Interpret activation functions as gradients of convex potentials to derive quadratic constraints.
• Formulate Lipschitz estimation as a semidefinite program (SDP) called LipSDP.
• Introduce single-layer and multi-layer LipSDP formulations with decision variables that control accuracy and scalability.
• Propose a hierarchy of relaxations (Neuron, Layer) and parallelization strategies for large networks.
• Provide a distributed implementation by splitting networks into sub-networks to preserve scalability.
• Discuss refinements that exploit repeated activation functions to tighten bounds when a fixed point is used.

Experimental results

Research questions

• RQ1 Can a convex SDP framework yield guaranteed upper bounds on the global Lipschitz constant of deep neural networks?
• RQ2 How can activation nonlinearities be encoded as quadratic constraints to enable scalable Lipschitz estimation?
• RQ3 What are the trade-offs between accuracy and scalability for LipSDP variants, and how do they perform on real datasets?
• RQ4 How does robust training affect Lipschitz bounds and robustness guarantees for classifiers?
• RQ5 Can the framework extend to multi-layer networks and other norms beyond the Euclidean norm?

Key findings

• LipSDP provides significantly tighter Lipschitz bounds than existing methods across tested networks.
• For MNIST-classifier networks, LipSDP bounds nearly match the true Lipschitz constant and outperform comparable methods.
• Robust training procedures (LP-Train, PGD-Train) reduce Lipschitz bounds, implying improved robustness.
• A distributed/sub-network approach markedly improves scalability for large networks without sacrificing bound quality.
• Empirical results show bounds can be used to derive non-vacuous lower bounds on minimum adversarial perturbations.