[Paper Review] Understanding Deep Neural Networks with Rectified Linear Units

Raman Arora, Amitabh Basu+2 more|arXiv (Cornell University)|Nov 4, 2016

Machine Learning and Algorithms33 references127 citations

TL;DR

The paper analyzes ReLU DNNs, proving exact correspondence with piecewise linear functions, giving a polynomial-time global optimizer for 1-hidden-layer networks, and establishing depth/size lower bounds and exponential growth of affine pieces via zonotopes.

ABSTRACT

In this paper we investigate the family of functions representable by deep neural networks (DNN) with rectified linear units (ReLU). We give an algorithm to train a ReLU DNN with one hidden layer to *global optimality* with runtime polynomial in the data size albeit exponential in the input dimension. Further, we improve on the known lower bounds on size (from exponential to super exponential) for approximating a ReLU deep net function by a shallower ReLU net. Our gap theorems hold for smoothly parametrized families of "hard" functions, contrary to countable, discrete families known in the literature. An example consequence of our gap theorems is the following: for every natural number $k$ there exists a function representable by a ReLU DNN with $k^2$ hidden layers and total size $k^3$, such that any ReLU DNN with at most $k$ hidden layers will require at least $\frac{1}{2}k^{k+1}-1$ total nodes. Finally, for the family of $\mathbb{R}^n o \mathbb{R}$ DNNs with ReLU activations, we show a new lowerbound on the number of affine pieces, which is larger than previous constructions in certain regimes of the network architecture and most distinctively our lowerbound is demonstrated by an explicit construction of a *smoothly parameterized* family of functions attaining this scaling. Our construction utilizes the theory of zonotopes from polyhedral theory.

Motivation & Objective

• Characterize the function class represented by ReLU DNNs and its relation to piecewise linear functions.
• Understand how depth and width affect expressivity and representational power.
• Establish lower bounds showing depth yields exponential/super-exponential benefits for certain functions.
• Provide constructs that achieve or bound the number of affine pieces in ReLU DNNs.
• Connect ReLU DNN expressivity to polyhedral geometry (zonotopes) and known approximation results.

Proposed method

• Show that every ReLU DNN computes a continuous piecewise linear function and that every continuous PWL function can be represented by a ReLU DNN with depth at most ceil(log2(n+1))+1.
• Provide constructive proofs for representing max operations and composing convex pieces within DNNs.
• Derive depth-size trade-offs and construct smoothly parameterized families of hard functions to separate shallow and deep networks.
• Use zonotopes and their support functions to bound the number of affine pieces and to build hard function families with provable complexity gaps.
• Present explicit theorems (e.g., Theorem 2.1, Theorem 2.3, Theorem 3.1, Corollaries 3.3–3.4) and supporting lemmas.

Experimental results

Research questions

• RQ1 What class of functions can ReLU DNNs represent and how does this relate to piecewise linear functions?
• RQ2 How do depth and width impact the expressive power and required size to represent certain functions?
• RQ3 Can we construct smoothly parameterized hard function families showing depth/size gaps for ReLU networks?
• RQ4 What are lower bounds on the number of affine pieces for ReLU DNNs and how do zonotopes contribute to these bounds?

Key findings

• ReLU DNNs compute exactly continuous piecewise linear functions, and every such function can be represented by a ReLU DNN with depth at most ceil(log2(n+1))+1 (Theorem 2.1).
• For n=1, a piecewise linear function with p pieces can be represented by a 2-layer DNN with at most p nodes, and any such DNN requires at least p-1 nodes (Theorem 2.2).
• Every Lq (1≤q≤∞) function on R^n can be arbitrarily well-approximated by a ReLU DNN with depth at most ceil(log2(n+1))+1 (Theorem 2.3).
• There exists a smoothly parameterized family of R→R hard functions representable by a (k+1)-layer ReLU DNN of width w for which any shallower network (≤k layers) must have size at least roughly (1/2) k' w^{k/k'} - 1 (Theorem 3.1).
• There are lower bounds showing the number of affine pieces grows exponentially in input dimension for certain R^n→R ReLU DNNs (Theorem 3.2 and related corollaries).
• The results provide super-exponential depth-size gaps and strengthen prior depth-separation work by using smooth parameterizations and zonotope-based constructions (Corollaries 3.3–3.4; Theorem 3.5).