[Paper Review] Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks

Fanghui Liu, Leello Dadi+1 more|arXiv (Cornell University)|Apr 29, 2024

Neural Networks and Applications4 citations

TL;DR

This paper establishes that norm-constrained, over-parameterized two-layer ReLU neural networks achieve improved sample complexity and generalization by leveraging the path norm (equivalent to the Barron norm), enabling width-independent convergence and metric entropy bounds of order $\mathcal{O}(\epsilon^{-2d/(d+2)})$, which outperforms kernel methods' $\Omega(\epsilon^{-d})$ scaling. It derives a generalization bound at rate $\mathcal{O}(n^{-(d+2)/(2d+2)})$ under a general moment hypothesis, demonstrating dimension-aware, improved sample efficiency.

ABSTRACT

Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks as the curse of dimensionality (CoD) cannot be evaded when trying to approximate even a single ReLU neuron (Bach, 2017). In this paper, we study a suitable function space for over-parameterized two-layer neural networks with bounded norms (e.g., the path norm, the Barron norm) in the perspective of sample complexity and generalization properties. First, we show that the path norm (as well as the Barron norm) is able to obtain width-independence sample complexity bounds, which allows for uniform convergence guarantees. Based on this result, we derive the improved result of metric entropy for $ε$-covering up to $O(ε^{-\frac{2d}{d+2}})$ ($d$ is the input dimension and the depending constant is at most linear order of $d$) via the convex hull technique, which demonstrates the separation with kernel methods with $Ω(ε^{-d})$ to learn the target function in a Barron space. Second, this metric entropy result allows for building a sharper generalization bound under a general moment hypothesis setting, achieving the rate at $O(n^{-\frac{d+2}{2d+2}})$. Our analysis is novel in that it offers a sharper and refined estimation for metric entropy with a linear dimension dependence and unbounded sampling in the estimation of the sample error and the output error.

Motivation & Objective

• To identify a suitable function space for over-parameterized two-layer neural networks with bounded norms, such as the path norm or Barron norm, to overcome limitations of reproducing kernel Hilbert spaces (RKHS) and kernel methods.
• To analyze the metric entropy of the function class induced by norm-constrained two-layer networks, especially under the path norm, to quantify its complexity and separation from kernel methods.
• To derive sharp generalization bounds under a general moment hypothesis, avoiding restrictive assumptions like sub-Gaussianity, and to show improved sample complexity scaling with respect to input dimension $d$.
• To demonstrate that the path norm enables width-independent generalization guarantees and avoids the curse of dimensionality inherent in kernel methods.

Proposed method

• The authors analyze the metric entropy of the function class defined by two-layer ReLU networks with bounded path norm using the convex hull technique, deriving an $\mathcal{O}(\epsilon^{-2d/(d+2)})$ covering number bound.
• They establish a generalization bound under a general moment hypothesis by combining sample error, output error, and regularization error analysis, using concentration inequalities and Gaussian process tools.
• The analysis leverages Bennett's and Wu et al.'s concentration inequalities to control the deviation of empirical risk from true risk, with explicit dependence on the input dimension $d$.
• The path norm is used as a capacity measure, and the function class is characterized as a subset of the Barron space, enabling dimension-aware generalization bounds.
• The proof framework integrates bounds on the output error via Rademacher complexity and regularization error via the path norm, with careful control of constants depending on $d$.
• A refined estimation of the metric entropy is achieved by exploiting the structure of the path norm and the activation function, leading to a tighter dependence on $d$ than previous results.

Experimental results

Research questions

• RQ1 Can the path norm or Barron norm provide a function space for two-layer neural networks that avoids the curse of dimensionality inherent in kernel methods?
• RQ2 What is the metric entropy of the function class induced by norm-constrained two-layer ReLU networks, and how does it scale with respect to the input dimension $d$?
• RQ3 Can generalization bounds be derived under a general moment hypothesis, rather than sub-Gaussian or bounded noise assumptions, and what is the resulting rate?
• RQ4 How does the sample complexity of norm-constrained two-layer networks compare to that of kernel methods in high-dimensional settings?
• RQ5 What is the dependence of the generalization error on the input dimension $d$, the sample size $n$, and the norm constraint $B$?

Key findings

• The metric entropy of the function class defined by path-norm-bounded two-layer ReLU networks is bounded by $\mathcal{O}(\epsilon^{-2d/(d+2)})$, which is significantly better than the $\Omega(\epsilon^{-d})$ scaling of kernel methods.
• The generalization error achieves a rate of $\mathcal{O}(n^{-(d+2)/(2d+2)})$ under a general moment hypothesis, which is faster than the $\Omega(n^{-1/d})$ minimax lower bound for kernel methods.
• The sample complexity is width-independent due to the path norm constraint, enabling uniform convergence guarantees across different network widths.
• The generalization bound is derived under a general moment condition, avoiding restrictive assumptions such as sub-Gaussianity or boundedness, and the bound explicitly depends on the input dimension $d$.
• The regularization error is controlled via the path norm, and the output error is bounded using concentration inequalities, with constants depending polynomially on $d$.
• The exponential decay term $\exp(-B/(4CM^2))$ in the bound ensures fast convergence for large norm constraints $B$, indicating strong generalization for well-regularized models.