[Paper Review] On the number of response regions of deep feed forward networks with piece-wise linear activations

Razvan Pascanu, Guido Montúfar+1 more|arXiv (Cornell University)|Dec 20, 2013

Advanced Memory and Neural Computing17 references126 citations

TL;DR

This paper analyzes the representational power of deep feedforward networks with ReLU activations by counting the number of linear regions—pieces of piecewise linear functions—defined in the input space. It shows that deep networks can achieve exponentially more linear regions than shallow networks with the same number of parameters, especially when depth increases, demonstrating a fundamental advantage in modeling complex functions through hierarchical composition.

ABSTRACT

This paper explores the complexity of deep feedforward networks with linear pre-synaptic couplings and rectified linear activations. This is a contribution to the growing body of work contrasting the representational power of deep and shallow network architectures. In particular, we offer a framework for comparing deep and shallow models that belong to the family of piecewise linear functions based on computational geometry. We look at a deep rectifier multi-layer perceptron (MLP) with linear outputs units and compare it with a single layer version of the model. In the asymptotic regime, when the number of inputs stays constant, if the shallow model has $kn$ hidden units and $n_0$ inputs, then the number of linear regions is $O(k^{n_0}n^{n_0})$. For a $k$ layer model with $n$ hidden units on each layer it is $Ω(\left\lfloor {n}/{n_0} ight floor^{k-1}n^{n_0})$. The number $\left\lfloor{n}/{n_0} ight floor^{k-1}$ grows faster than $k^{n_0}$ when $n$ tends to infinity or when $k$ tends to infinity and $n \geq 2n_0$. Additionally, even when $k$ is small, if we restrict $n$ to be $2n_0$, we can show that a deep model has considerably more linear regions that a shallow one. We consider this as a first step towards understanding the complexity of these models and specifically towards providing suitable mathematical tools for future analysis.

Motivation & Objective

• To understand why deep neural networks can represent complex functions more efficiently than shallow networks.
• To quantify the representational capacity of deep feedforward networks with piecewise linear activations like ReLU.
• To compare the number of linear regions in deep versus shallow architectures under fixed parameter budgets.
• To develop a geometric framework based on hyperplane arrangements to analyze the complexity of deep networks.

Proposed method

• Modeling deep ReLU networks as compositions of piecewise linear functions to analyze their response regions.
• Using computational geometry to count the number of linear regions formed by hyperplane arrangements in the input space.
• Deriving upper and lower bounds on the number of linear regions for shallow and deep architectures using combinatorial sums.
• Applying asymptotic analysis (Big-O and Big-Omega notation) to compare growth rates of linear regions with respect to depth and width.
• Relating the number of linear regions to the number of parameters to assess representational efficiency.
• Establishing that deep models grow exponentially in region count with depth, while shallow models grow polynomially.

Experimental results

Research questions

• RQ1 How does the number of linear regions in a deep ReLU network scale with depth and width compared to a shallow network with the same number of parameters?
• RQ2 Can deep networks achieve exponentially more linear regions than shallow networks when parameter counts are fixed?
• RQ3 What is the relationship between the number of linear regions and the depth of a ReLU network with fixed input dimension?
• RQ4 How does the number of linear regions grow as a function of the number of parameters in deep versus shallow architectures?
• RQ5 To what extent does hierarchical composition in deep networks increase representational capacity beyond what shallow networks can achieve?

Key findings

• For a shallow network with $ kn $ hidden units and $ n_0 $ inputs, the number of linear regions is $ O(k^{n_0}n^{n_0}) $ when $ n_0 = O(1) $.
• For a $ k $-layer deep network with $ n $ hidden units per layer and $ n_0 $ inputs, the number of linear regions is $ \Omega\left(\left\lfloor\frac{n}{n_0}\right\rfloor^{k-1}n^{n_0}\right) $ when $ n_0 = O(1) $.
• When $ n \geq 2n_0 $, the deep model's region count grows faster than the shallow model's as $ n \to \infty $ or $ k \to \infty $.
• With $ n = 2n_0 $, even small $ k $, the deep model has significantly more linear regions than the shallow model.
• The ratio of linear regions to parameters grows exponentially with depth $ k $, indicating superior representational efficiency in deep models.
• The number of parameters in a deep model is $ O(kn^2) $, while in a shallow model it is $ O(kn) $, showing that deep models achieve more regions per parameter.