[Paper Review] The Power of Depth for Feedforward Neural Networks

Ronen Eldan, Ohad Shamir|arXiv (Cornell University)|Dec 12, 2015

Machine Learning and Algorithms18 references215 citations

TL;DR

This paper demonstrates that a simple radial function on ℝᵈ, computable by a 3-layer feedforward neural network of small width, cannot be approximated by any 2-layer network to better than a constant accuracy unless the width is exponential in the input dimension d. The result holds for all standard activation functions (ReLU, sigmoid, threshold), formally establishing that depth provides an exponential advantage over width in expressive capacity for fully connected feedforward networks.

ABSTRACT

We show that there is a simple (approximately radial) function on $ eals^d$, expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, to more than a certain constant accuracy, unless its width is exponential in the dimension. The result holds for virtually all known activation functions, including rectified linear units, sigmoids and thresholds, and formally demonstrates that depth -- even if increased by 1 -- can be exponentially more valuable than width for standard feedforward neural networks. Moreover, compared to related results in the context of Boolean functions, our result requires fewer assumptions, and the proof techniques and construction are very different.

Motivation & Objective

• To investigate whether depth provides a fundamental advantage over width in the expressive power of feedforward neural networks.
• To formally establish that a 3-layer network can compute functions that require exponentially wide 2-layer networks to approximate.
• To analyze the expressive capacity of shallow versus deep networks under bounded width and depth constraints.
• To demonstrate that depth, even by just one layer, can yield an exponential advantage over width for standard activation functions.
• To provide a general result that applies to a wide class of activation functions, including ReLU, sigmoid, and threshold functions.

Proposed method

• Constructs a radial function f(‖x‖) on ℝᵈ that is 1-Lipschitz and compactly supported, using a sum of signed, scaled, and shifted ReLU-like neurons.
• Uses a two-stage approximation: first approximating a univariate function via a 2-layer network with bounded width, then lifting this to a radial function in d dimensions.
• Employs a kernel-based construction to create a function g(x) = ∑ᵢ εᵢ gᵢ(‖x‖) where each gᵢ is a 3-layer network, ensuring the sum is not approximable by 2-layer networks.
• Applies a universal approximation property for 2-layer networks (Assumption 1) to show that any 2-layer network approximating the constructed function must have width exponential in d.
• Uses the triangle inequality in L₂(μ) space to combine approximation errors from two stages: one for the sum of signed functions and one for the radial function approximation.
• Derives a lower bound on the L₂(μ) distance between any 2-layer network and the constructed 3-layer function, proving that no such 2-layer network can approximate it within a constant error unless width is exponential.

Experimental results

Research questions

• RQ1 Can a 3-layer feedforward neural network compute functions that cannot be approximated by any 2-layer network, even with arbitrarily large width?
• RQ2 What is the minimal width required for a 2-layer network to approximate a function expressible by a 3-layer network with small width?
• RQ3 Does depth provide an exponential advantage over width in the expressive capacity of feedforward networks?
• RQ4 Does this depth advantage hold for all standard activation functions, including ReLU, sigmoid, and threshold functions?
• RQ5 Can a radial function on ℝᵈ be constructed such that its 3-layer network representation is provably unapproximable by 2-layer networks without exponential width?

Key findings

• There exists a radial function f(‖x‖) on ℝᵈ that can be computed by a 3-layer feedforward network of constant width (independent of d), but cannot be approximated by any 2-layer network to within a constant error unless the width is exponential in d.
• The lower bound on approximation error holds for all standard activation functions satisfying Assumption 1, including ReLU, sigmoid, and threshold functions.
• The required width of any 2-layer network approximating the constructed 3-layer function grows as exp(Ω(d)), demonstrating an exponential separation in width between 2-layer and 3-layer networks.
• The proof constructs a function g(x) expressible by a 3-layer network of width at most C′cσ d¹⁹ᐟ⁴, where cσ is a constant depending on the activation function, and C′ is a universal constant.
• The L₂(μ) distance between any 2-layer network and the constructed 3-layer function is bounded below by δ₁/2 > 0, proving that no 2-layer network can approximate it within this error unless width is exponential.
• The result holds under minimal assumptions on the activation function, requiring only that it satisfies the universal approximation property for univariate Lipschitz functions on bounded domains.