[Paper Review] Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation

Colin Wei, Tengyu Ma|arXiv (Cornell University)|May 9, 2019

Sparse and Compressive Sensing TechniquesEngineering53 references45 citations

TL;DR

The paper develops data-dependent Rademacher complexity bounds for deep networks by augmenting networks with Lipschitz-regularizing properties (layer norms and Jacobian norms) and shows polynomial-in-depth generalization bounds; it also proposes Jacobian-based regularization during training.

ABSTRACT

Existing Rademacher complexity bounds for neural networks rely only on norm control of the weight matrices and depend exponentially on depth via a product of the matrix norms. Lower bounds show that this exponential dependence on depth is unavoidable when no additional properties of the training data are considered. We suspect that this conundrum comes from the fact that these bounds depend on the training data only through the margin. In practice, many data-dependent techniques such as Batchnorm improve the generalization performance. For feedforward neural nets as well as RNNs, we obtain tighter Rademacher complexity bounds by considering additional data-dependent properties of the network: the norms of the hidden layers of the network, and the norms of the Jacobians of each layer with respect to all previous layers. Our bounds scale polynomially in depth when these empirical quantities are small, as is usually the case in practice. To obtain these bounds, we develop general tools for augmenting a sequence of functions to make their composition Lipschitz and then covering the augmented functions. Inspired by our theory, we directly regularize the network's Jacobians during training and empirically demonstrate that this improves test performance.

Motivation & Objective

• Motivate why data-dependent properties beyond margin can tighten generalization bounds for deep networks.
• Develop a framework to bound Rademacher complexity by augmenting networks with data-dependent Lipschitz properties.
• Derive a polynomial-in-depth generalization bound that depends on hidden-layer norms and interlayer Jacobian norms on the training data.
• Propose a practical regularization strategy that controls Jacobians during training and evaluate its impact on test performance.

Proposed method

• Represent neural networks as compositions of layer-operators and activations.
• Introduce a Lipschitz augmentation of the loss by soft indicators on layer norms and Jacobians to bound worst-case Lipschitz constants.
• Extend covering-number techniques to computational-graph representations to handle augmented losses.
• Derive a bound (simplified form) showing L-test <= tilde-O of quantities involving layer norms, Jacobian norms, width-independent terms, and depth-r, on training data.
• Provide a formal framework (Theorem 4.3, 5.3, 6.x) for covering augmented computational graphs and obtaining data-dependent generalization bounds.

Experimental results

Research questions

• RQ1 What data-dependent network properties (beyond margin) can tighten generalization bounds for deep nets?
• RQ2 Can Lipschitz augmentation and Jacobian norms yield bounds that scale polynomially with depth rather than exponentially?
• RQ3 How can covering-number arguments be extended to augmented computational graphs to bound data-dependent losses?
• RQ4 Does regularizing the network’s Jacobians improve empirical generalization performance?

Key findings

• A data-dependent generalization bound is obtained that scales polynomially in depth when the training-data Lipschitz constants and Jacobian norms are small.
• The bound depends on the maximum hidden-layer norms t and the maximum interlayer Jacobian operator norms sigma measured on training data.
• Augmenting the loss with soft indicators for layer norms and Jacobians yields a tractable Lipschitz-bound framework for covering numbers.
• The authors provide a practical interpretation: regularizing Jacobians during training can improve test performance in experiments.
• The framework extends to recurrent nets with analogous polynomial-in-depth bounds.