[Paper Review] Representation Benefits of Deep Feedforward Networks

Matus Telgarsky|arXiv (Cornell University)|Sep 27, 2015

Neural Networks and Applications5 references141 citations

TL;DR

This paper demonstrates that deep feedforward ReLU networks can solve certain classification problems with zero error using exponentially fewer parameters than shallow networks, which are forced into at least 1/6 error. It constructs a family of problems (n-ap) with alternating labels on 2^k uniformly spaced points, showing that a 2-node recurrent network with 2k layers achieves perfect classification, while shallow networks require exponentially more nodes to match this performance.

ABSTRACT

This note provides a family of classification problems, indexed by a positive integer $k$, where all shallow networks with fewer than exponentially (in $k$) many nodes exhibit error at least $1/6$, whereas a deep network with 2 nodes in each of $2k$ layers achieves zero error, as does a recurrent network with 3 distinct nodes iterated $k$ times. The proof is elementary, and the networks are standard feedforward networks with ReLU (Rectified Linear Unit) nonlinearities.

Motivation & Objective

• To demonstrate that deep feedforward networks can achieve perfect classification on specific problems where shallow networks fail.
• To quantify the exponential advantage of depth in representation power using a concrete classification problem with alternating labels.
• To show that even recurrent networks with few parameters can achieve zero error on the same problem, highlighting architectural efficiency.
• To provide an elementary proof of representation benefits using sawtooth function analysis and counting arguments on function complexity.
• To connect the results to broader themes in neural network expressivity, circuit complexity, and statistical learning theory.

Proposed method

• Constructs the n-alternating-point (n-ap) problem with 2^k uniformly spaced points in [0,1] and alternating binary labels.
• Uses a piecewise affine function f_m^k that maps each point x_i to its label y_i exactly, via iterative composition of a 2-node ReLU network.
• Proves that any shallow network with m nodes per layer and l layers can produce at most (tm)^l sawtooth regions when σ is t-sawtooth.
• Applies a counting argument: a t-sawtooth function can cross 1/2 at most 2t times, limiting its ability to match rapidly alternating labels.
• Uses the fact that f_m^k is a 2^k-sawtooth function, allowing it to correctly classify all 2^k points.
• Establishes that the recurrent network f_m^k ∈ R(σ_r;2,2;k) achieves zero error, while shallow networks with fewer than exponentially many nodes cannot.

Experimental results

Research questions

• RQ1 Can deep feedforward networks achieve zero classification error on a problem where shallow networks are fundamentally limited?
• RQ2 What is the minimal number of nodes required in a shallow network to match the performance of a deep network on a structured classification problem?
• RQ3 How does the representation power of recurrent networks compare to standard feedforward networks in terms of parameter efficiency?
• RQ4 To what extent do sawtooth function properties constrain the classification error of shallow networks?
• RQ5 What is the quantitative trade-off between depth, width, and number of parameters in achieving exact classification on structured data?

Key findings

• For the 2^k-alternating-point problem, a deep network with 2 nodes in each of 2k layers achieves zero classification error.
• Any shallow network with m nodes per layer and l layers must have classification error at least (n - 4(2m)^l)/(3n), where n = 2^k.
• When m ≤ 2^{(k-3)/l - 1}, the minimum error of shallow networks is bounded below by 1/6, even as k grows.
• A recurrent network with only 2 nodes per layer, iterated k times, achieves zero error on the same problem.
• The number of regions (sawtooth pieces) in a function from a shallow network is bounded by (tm)^l, which limits its ability to match rapidly changing labels.
• The result shows an exponential separation in parameter efficiency: shallow networks require exponentially more nodes than deep or recurrent networks to achieve zero error.