[Paper Review] The Efficacy of $L_1$ Regularization in Two-Layer Neural Networks

Gen Li, Yuantao Gu+1 more|arXiv (Cornell University)|Oct 2, 2020

Neural Networks and Applications4 citations

TL;DR

This paper demonstrates that $L_1$ regularization in two-layer neural networks enables near-minimax optimal generalization error bounds by controlling model complexity without explicit neuron count selection. By applying $L_1$ regularization to the output layer coefficients, the risk bound achieves $O((d/n)^{1/2})$ rate (up to log factors), while input-layer $L_1$ regularization yields a dimension-free bound that does not scale with input dimension $d$, enabling sparsity and robustness in high-dimensional settings.

ABSTRACT

A crucial problem in neural networks is to select the most appropriate number of hidden neurons and obtain tight statistical risk bounds. In this work, we present a new perspective towards the bias-variance tradeoff in neural networks. As an alternative to selecting the number of neurons, we theoretically show that $L_1$ regularization can control the generalization error and sparsify the input dimension. In particular, with an appropriate $L_1$ regularization on the output layer, the network can produce a statistical risk that is near minimax optimal. Moreover, an appropriate $L_1$ regularization on the input layer leads to a risk bound that does not involve the input data dimension. Our analysis is based on a new amalgamation of dimension-based and norm-based complexity analysis to bound the generalization error. A consequent observation from our results is that an excessively large number of neurons do not necessarily inflate generalization errors under a suitable regularization.

Motivation & Objective

• To address the challenge of model complexity and statistical risk in two-layer neural networks, particularly the difficulty of selecting the optimal number of hidden neurons.
• To investigate whether $L_1$ regularization can serve as an alternative to explicit architecture selection for controlling generalization error.
• To derive tight statistical risk bounds that are robust to high input dimensionality and avoid the bias-variance tradeoff associated with neuron count.
• To establish that an excessively large number of neurons does not necessarily increase generalization error under proper $L_1$ regularization.
• To develop a unified complexity analysis combining dimension-based and norm-based approaches for generalization error bounds.

Proposed method

• Proposes a constrained $L_1$ estimator over a function class with $\|a\|_1 \leq V$ for output weights and $\sup_j (\|w_j\|_1 + |b_j|) \leq \eta$ for input weights, enabling joint regularization.
• Introduces a novel amalgamation of dimension-based and norm-based complexity analysis to bound generalization error, combining Rademacher complexity and norm-based measures.
• Derives risk bounds under $L_1$ training loss, showing convergence rates approaching the minimax optimal $O((d/n)^{1/2})$ under appropriate regularization.
• Imposes $L_1$ constraints on input-layer weights to achieve a dimension-free risk bound that depends only on the true sparsity level $k$, not $d$.
• Uses a bounded input domain assumption to derive a risk bound independent of $d$ and $r$, with $\mathcal{R}(\hat{f}_n) \lesssim C\delta_\eta + \frac{V\eta + \tau}{\sqrt{n}}$.
• Employs a ReLU or sigmoid activation function and assumes sub-Gaussian noise to derive high-probability risk bounds via concentration inequalities.

Experimental results

Research questions

• RQ1 Can $L_1$ regularization on the output layer achieve a statistical risk bound that is near minimax optimal, approaching the parametric rate $O((d/n)^{1/2})$?
• RQ2 Does $L_1$ regularization on the input layer lead to a risk bound independent of the input dimension $d$, thereby enabling sparsity in high-dimensional settings?
• RQ3 Can a single large neural network with $L_1$ regularization outperform model selection via multiple architectures with varying neuron counts?
• RQ4 Does an excessively large number of hidden neurons increase generalization error when combined with $L_1$ regularization?
• RQ5 Can the proposed method achieve dimension-free risk bounds in high-dimensional input regimes where $d \gg n$?

Key findings

• The $L_1$ regularized estimator on the output layer achieves a risk bound of $\mathcal{R}(\hat{f}_n) \lesssim \sqrt{\{d\log(dn)\}/n}$, approaching the minimax optimal rate $O((d/n)^{1/2})$ under $L_1$ training loss.
• With $L_1$ regularization on the input layer, the risk bound becomes $\mathcal{R}(\hat{f}_n) \lesssim \sqrt{\{k\log(dn)\}/n}$, where $k$ is the true sparsity level, independent of $d$.
• When both output and input layers are regularized, the risk bound $\mathcal{R}(\hat{f}_n) \lesssim C\delta_\eta + \frac{V\eta + \tau}{\sqrt{n}}$ becomes independent of $d$ and $r$ for large $r$.
• For $\sigma(x) = 1/(1+e^{-x})$ and $\eta \asymp (n\log^2 n)^{1/3}$, the risk bound achieves $\mathcal{R}(\hat{f}_n) \lesssim V(\log n / n)^{1/3}$, which is tighter than $O(n^{-1/2})$ when $d \gg n$.
• The analysis shows that a large number of neurons does not necessarily increase generalization error under $L_1$ regularization, as the regularization controls the effective model complexity.
• The proposed method achieves a dimension-free risk bound that is minimax optimal up to logarithmic factors, even when $d \gg n$.