[Paper Review] Why Deep Neural Networks for Function Approximation?

Shiyu Liang, R. Srikant|arXiv (Cornell University)|Oct 13, 2016

Neural Networks and ApplicationsComputer Science152 citations

TL;DR

The paper shows that for a large class of piecewise smooth functions, deep networks require exponentially fewer neurons than shallow ones to achieve a given approximation accuracy.

ABSTRACT

Recently there has been much interest in understanding why deep neural networks are preferred to shallow networks. We show that, for a large class of piecewise smooth functions, the number of neurons needed by a shallow network to approximate a function is exponentially larger than the corresponding number of neurons needed by a deep network for a given degree of function approximation. First, we consider univariate functions on a bounded interval and require a neural network to achieve an approximation error of $\varepsilon$ uniformly over the interval. We show that shallow networks (i.e., networks whose depth does not depend on $\varepsilon$) require $Ω( ext{poly}(1/\varepsilon))$ neurons while deep networks (i.e., networks whose depth grows with $1/\varepsilon$) require $\mathcal{O}( ext{polylog}(1/\varepsilon))$ neurons. We then extend these results to certain classes of important multivariate functions. Our results are derived for neural networks which use a combination of rectifier linear units (ReLUs) and binary step units, two of the most popular type of activation functions. Our analysis builds on a simple observation: the multiplication of two bits can be represented by a ReLU.

Motivation & Objective

• Motivate why depth helps in function approximation for neural networks.
• Establish upper and lower bounds on network size needed for epsilon-approximation across function classes.
• Compare depth versus width for univariate and multivariate functions using ReLU and binary step activations.

Proposed method

• Provide L-infinity (uniform) approximation bounds on function classes by neural networks.
• Use constructions that implement binary expansions and polynomial evaluations via ReLUs and BSUs.
• Derive upper bounds: deep networks with O(log(1/epsilon)) layers and polylog(1/epsilon) neurons; shallow nets require poly(1/epsilon) neurons.
• Prove lower bounds for strongly convex functions showing N >= log(1/epsilon) necessary.

Experimental results

Research questions

• RQ1 What depth and size suffice to achieve an epsilon-approximation of a given function?
• RQ2 For a fixed depth, what is the minimum network size needed to achieve epsilon-approximation?
• RQ3 How do depth and size trade off for univariate and multivariate functions under ReLU and binary step activations?
• RQ4 Are the derived bounds tight for broad function classes, including piecewise smooth and strongly convex functions?
• RQ5 How do results extend to compositions, products, and linear combinations of functions?

Key findings

• Shallow networks require Omega(poly(1/epsilon)) neurons to achieve epsilon-approximation on bounded intervals for piecewise smooth univariate functions.
• Deep networks with depth Theta(log(1/epsilon)) require O(polylog(1/epsilon)) neurons for the same epsilon-approximation.
• For differentiable and strongly convex functions, any multilayer network needs Omega(log(1/epsilon)) neurons to achieve epsilon-approximation.
• Corollaries show polylog(1/epsilon) bounds extend to additions, multiplications, and compositions of smooth univariate or multivariate functions.
• Lower bounds imply the depth-based advantages are tight under the stated assumptions.