[Paper Review] The power of deeper networks for expressing natural functions

David Rolnick, Max Tegmark|arXiv (Cornell University)|May 16, 2017

Neural Networks and Applications16 references90 citations

TL;DR

The paper proves that deep feedforward networks express natural polynomials much more efficiently than shallow ones, with neuron counts growing linearly in n for deep nets vs exponentially for single-hidden-layer nets; increasing layers yields exponential savings roughly scaling with n^{1/k}.

ABSTRACT

It is well-known that neural networks are universal approximators, but that deeper networks tend in practice to be more powerful than shallower ones. We shed light on this by proving that the total number of neurons $m$ required to approximate natural classes of multivariate polynomials of $n$ variables grows only linearly with $n$ for deep neural networks, but grows exponentially when merely a single hidden layer is allowed. We also provide evidence that when the number of hidden layers is increased from $1$ to $k$, the neuron requirement grows exponentially not with $n$ but with $n^{1/k}$, suggesting that the minimum number of layers required for practical expressibility grows only logarithmically with $n$.

Motivation & Objective

• Motivate theoretical understanding of why deep networks can outperform shallow ones in representing natural functions.
• Quantify resource requirements (neurons) for approximating multivariate polynomials with networks of varying depth.
• Show that depth dramatically reduces required neurons for sparse polynomials and natural functions.
• Explore how the minimum number of layers needed grows with input dimension n and polynomial sparsity.

Proposed method

• Model neural networks as N(x)=A_k σ(... σ(A_0 x)) with depth k.
• Use epsilon-uniform approximation versus Taylor approximation to analyze expressivity.
• Prove that for a degree-d polynomial with nonzero Taylor coefficients up to degree d, the minimal neurons m^ε_k(p) converges to a finite limit as ε→0.
• Derive upper bounds showing m^uniform_1(p) = product_i (r_i+1) for monomials, while m^uniform(p) scales as sum_i 7⌈log2(r_i)⌉+4.
• Extend analysis to univariate and multivariate polynomials and to sparse polynomials with c monomials.
• Propose a Taylor-based constructive method to bound depth-k representations of products, yielding m^uniform_k(p) = O(n^{(k-1)/k} 2^{n^{1/k}}) for p = x_1 x_2 ... x_n.

Experimental results

Research questions

• RQ1 How does the neuron count required to approximate multivariate polynomials depend on network depth?
• RQ2 Is there an exponential gap in efficiency between shallow and deep networks for natural polynomial functions?
• RQ3 How does the minimum required number of layers scale with input dimension n and polynomial sparsity?
• RQ4 Can sparse polynomials be represented with significantly fewer neurons in deeper architectures compared to shallow ones?
• RQ5 What are concrete constructive bounds for depth-k networks approximating products of inputs?

Key findings

• For a degree-d multivariate polynomial with nonzero Taylor coefficients up to degree d, m^ε_1(p) ≤ ∏_i (r_i+1).
• For the same polynomial, m^uniform(p) ≤ ∑_i (7⌈log2(r_i)⌉+4).
• There is an exponential gap between shallow and deep representations for monomials and sparse polynomials under uniform (and Taylor) approximation.
• For univariate degree-d polynomials, m^Taylor_1(p) ≤ d+1, while m^uniform(p) can be kept logarithmic in d.
• For p = x1 x2 ... xn, m^uniform_k(p) = O(n^{(k-1)/k} 2^{n^{1/k}}), demonstrating that increasing depth dramatically reduces resource needs.
• Empirically, the bound n^{1/k} guides how many layers are needed to keep layer width around feasible sizes (e.g., 2^{10}).