[Paper Review] Empirical Risk Minimization in the Interpolating Regime with Application to Neural Network Learning

Nicole Mücke, Ingo Steinwart|arXiv (Cornell University)|May 25, 2019

Model Reduction and Neural Networks26 references4 citations

TL;DR

This paper establishes theoretical guarantees for empirical risk minimization (ERM) in the interpolating regime—where models achieve zero training error—showing that some ERM methods over large hypothesis classes are universally consistent and achieve optimal learning rates, while others fail catastrophically. The authors further demonstrate that similar behavior occurs in deep ReLU networks with sufficient width, proving consistency and near-optimal convergence rates under mild conditions.

ABSTRACT

A common strategy to train deep neural networks (DNNs) is to use very large architectures and to train them until they (almost) achieve zero training error. Empirically observed good generalization performance on test data, even in the presence of lots of label noise, corroborate such a procedure. On the other hand, in statistical learning theory it is known that over-fitting models may lead to poor generalization properties, occurring in e.g. empirical risk minimization (ERM) over too large hypotheses classes. Inspired by this contradictory behavior, so-called interpolation methods have recently received much attention, leading to consistent and optimally learning methods for some local averaging schemes with zero training error. However, there is no theoretical analysis of interpolating ERM-like methods so far. We take a step in this direction by showing that for certain, large hypotheses classes, some interpolating ERMs enjoy very good statistical guarantees while others fail in the worst sense. Moreover, we show that the same phenomenon occurs for DNNs with zero training error and sufficiently large architectures.

Motivation & Objective

• To resolve the paradox of good generalization in overparameterized deep neural networks (DNNs) despite zero training error.
• To analyze whether empirical risk minimization (ERM) over large hypothesis classes can yield consistent and optimal learning performance.
• To identify conditions under which interpolating ERM methods succeed or fail, even when achieving zero training error.
• To extend theoretical analysis from classical nonparametric methods to DNNs with zero training error.
• To show that specific DNN architectures with linearly growing widths in input dimension and sample size achieve consistency and near-optimal learning rates.

Proposed method

• Constructs large hypothesis classes $\mathcal{H}_n$ for which some interpolating least squares ERM algorithms are universally consistent, while others converge to the negative regression function.
• Uses data-dependent cubic partitioning rules $\pi_{m_n,s_n}$ to define histogram-based predictors $h_{D,\mathcal{A}_D}$, where $\mathcal{A}_D$ is a partition of the input space based on training data.
• Applies covering number and concentration inequalities to bound excess risk, leveraging $\varepsilon$-covering numbers of $\mathcal{H}_\mathcal{A}$ and $|\mathcal{P}_n| \leq c n^{1+\beta}$.
• Derives learning rates for the empirical histogram rule under Hölder continuity of the Bayes function, using $s_n = n^{-\gamma}$ with $\gamma = 1/(2\alpha + d)$.
• Proves that for $s_n \to 0$ and $\ln(ns_n^d)/(ns_n^d) \to 0$, excess risk converges to zero in probability.
• Extends results to ReLU DNNs with at least two hidden layers, showing consistency and $\mathcal{O}(d^2 n^2)$ training time for fully connected networks.

Experimental results

Research questions

• RQ1 Can empirical risk minimization over large hypothesis classes still yield consistent and optimal learning performance when achieving zero training error?
• RQ2 What distinguishes interpolating ERM methods that generalize well from those that fail catastrophically?
• RQ3 Do the same statistical phenomena observed in classical nonparametric methods (e.g., histogram rules) also occur in deep neural networks with zero training error?
• RQ4 Can deep ReLU networks with widths growing linearly in $n$ and $d$ achieve consistency and near-optimal learning rates in the interpolating regime?
• RQ5 What conditions on data partitioning and network architecture ensure that interpolating predictors converge to the Bayes risk?

Key findings

• Some interpolating ERM methods over large hypothesis classes are universally consistent and achieve excess risk bounds decaying as $\mathcal{O}(\ln(n)/n)^{2\alpha\gamma}$ with high probability.
• Other interpolating ERM methods fail completely, converging to the negative of the regression function and achieving risks far from optimal.
• For $\alpha$-Hölder continuous Bayes functions, the proposed histogram-based ERM achieves learning rates matching known optimal rates up to a $\ln(n)$ factor.
• The excess risk converges to zero in probability as $n \to \infty$, provided $\ln(ns_n^d)/(ns_n^d) \to 0$ and $s_n \to 0$.
• ReLU DNNs with at least two hidden layers and widths growing linearly in $d$ and $n$ achieve consistency and near-optimal rates under the same assumptions.
• Training such DNNs can be performed in $\mathcal{O}(d^2 n^2)$ time when implemented as fully connected networks.