[Paper Review] Approximation and Estimation for High-Dimensional Deep Learning Networks

Andrew R. Barron, Jason M. Klusowski|arXiv (Cornell University)|Sep 10, 2018

Machine Learning and Algorithms35 references42 citations

TL;DR

The paper derives risk (mean squared error) bounds for deep ramp networks with L1-type weight controls, showing minimax-like rates that depend on log d and depth L, not directly on parameter count.

ABSTRACT

It has been experimentally observed in recent years that multi-layer artificial neural networks have a surprising ability to generalize, even when trained with far more parameters than observations. Is there a theoretical basis for this? The best available bounds on their metric entropy and associated complexity measures are essentially linear in the number of parameters, which is inadequate to explain this phenomenon. Here we examine the statistical risk (mean squared predictive error) of multi-layer networks with $\ell^1$-type controls on their parameters and with ramp activation functions (also called lower-rectified linear units). In this setting, the risk is shown to be upper bounded by $[(L^3 \log d)/n]^{1/2}$, where $d$ is the input dimension to each layer, $L$ is the number of layers, and $n$ is the sample size. In this way, the input dimension can be much larger than the sample size and the estimator can still be accurate, provided the target function has such $\ell^1$ controls and that the sample size is at least moderately large compared to $L^3\log d$. The heart of the analysis is the development of a sampling strategy that demonstrates the accuracy of a sparse covering of deep ramp networks. Lower bounds show that the identified risk is close to being optimal.

Motivation & Objective

• Motivate and quantify why deep networks generalize well in high-dimensional settings with more parameters than samples.
• Introduce and formalize variation and average variation notions for multi-layer networks to capture complexity.
• Develop sparse approximants and covering number bounds to balance estimation error and model complexity.
• Establish risk bounds for networks under L1-type weight controls and ramp activations.
• Demonstrate near-optimal minimax rates under the proposed framework.

Proposed method

• Model deep networks with ramp activations and nonnegative (or sign-handled) weights.
• Define network variation V_L and subnetwork variations V_j^out, V_j^in, and average variation 1overline{V} to quantify size.
• Express f(W,x) via a product-structured weight representation and introduce a Markov-like decomposition a_{j1,...,jL} of the weights.
• Construct sparse approximants by random-representer covers of fixed cardinality M, yielding a bound on covering numbers.
• Prove main risk bound: for composite variation v = 1overline{V} sqrt{V}, the squared error scales as (L v / sqrt{M})^2 under an appropriate probability measure.

Experimental results

Research questions

• RQ1 What are the theoretical risk guarantees for deep networks with ramp activations when parameter norms are controlled?
• RQ2 How can one quantify and leverage network variation to enable sparse approximations and favorable generalization bounds?
• RQ3 Can we construct sparse network approximants with provable covering number bounds that yield minimax-like rates?
• RQ4 How do depth L and input dimension d influence the learning risk under L1-type penalization?

Key findings

• The risk bound is upper bounded by [(L^3 log d)/n]^{1/2} for the examined class, enabling accurate estimation even when d is large relative to n, given suitable L and log d factors.
• A sparse covering argument yields a subfamily with log-cardinality at most (L-2)M log(min{d_bar, 2M}) + M log(8e d_in).
• The main theorem shows an error bound for any f(W,x) in the class with composite variation v = overline{V} sqrt{V}, demonstrating near-minimax rates under the proposed framework.
• Lower bounds indicate the identified risk is close to optimal within the defined model class.
• Representability and conservation-like canonical forms balance interlayer weight flow to facilitate analysis and tighten bounds.
• The approach emphasizes variation-based complexity control rather than parameter-count-based measures, addressing high-dimensional generalization phenomena.