[Paper Review] The Expressive Power of Neural Networks: A View from the Width

Lu Zhou, Hongming Pu+3 more|arXiv (Cornell University)|Sep 8, 2017

Advanced Memory and Neural Computing7 references125 citations

TL;DR

Width-bounded ReLU networks can universal-approximate any Lebesgue-integrable function with width n+4, while width n cannot, revealing a width-based phase transition; the paper also establishes polynomial lower bounds for width efficiency and provides supportive experiments.

ABSTRACT

The expressive power of neural networks is important for understanding deep learning. Most existing works consider this problem from the view of the depth of a network. In this paper, we study how width affects the expressiveness of neural networks. Classical results state that depth-bounded (e.g. depth-$2$) networks with suitable activation functions are universal approximators. We show a universal approximation theorem for width-bounded ReLU networks: width-$(n+4)$ ReLU networks, where $n$ is the input dimension, are universal approximators. Moreover, except for a measure zero set, all functions cannot be approximated by width-$n$ ReLU networks, which exhibits a phase transition. Several recent works demonstrate the benefits of depth by proving the depth-efficiency of neural networks. That is, there are classes of deep networks which cannot be realized by any shallow network whose size is no more than an exponential bound. Here we pose the dual question on the width-efficiency of ReLU networks: Are there wide networks that cannot be realized by narrow networks whose size is not substantially larger? We show that there exist classes of wide networks which cannot be realized by any narrow network whose depth is no more than a polynomial bound. On the other hand, we demonstrate by extensive experiments that narrow networks whose size exceed the polynomial bound by a constant factor can approximate wide and shallow network with high accuracy. Our results provide more comprehensive evidence that depth is more effective than width for the expressiveness of ReLU networks.

Motivation & Objective

• Investigate how the width of ReLU networks affects expressive power beyond the well-studied depth perspective.
• Prove a universal approximation theorem for width-bounded networks and identify a width threshold (n+4) for L1 approximation on R^n.
• Examine width efficiency by establishing polynomial lower bounds for approximating wide networks with narrow ones.
• Provide experimental evidence regarding practical width-depth tradeoffs and their implications for network design.

Proposed method

• Construct a width-(n+4) fully-connected ReLU network that approximates any Lebesgue-integrable function with arbitrary L1 error ε.
• Decompose target functions into finite sums of indicator functions on axis-aligned cubes and approximate these indicators with ReLU-based blocks.
• Introduce a block-wise network architecture that stores and sums approximations across cubes to build up the global approximation.
• Prove a width-bounded universal approximation theorem (Theorem 1) via a constructive network design, and compare with classical depth-bounded universal approximation.
• Analyze width efficiency by deriving a polynomial lower bound (Theorem 4) for approximating wide networks with narrower ones, and discuss experimental verification.

Experimental results

Research questions

• RQ1 Does width-bounded ReLU networks with width n+4 universal-approximate Lebesgue-integrable functions in R^n under L1 distance?
• RQ2 Is there a phase transition in expressiveness when width crosses the threshold n to n+1?
• RQ3 Do there exist wide networks that cannot be approximated by narrow networks unless the latter have polynomially larger size?
• RQ4 Do experimental results support a polynomial (rather than exponential) trade-off between width and required network size for approximation?

Key findings

• Width-(n+4) ReLU networks can approximate any Lebesgue-integrable function on R^n to arbitrary L1 accuracy.
• Except for a measure-zero set, functions cannot be approximated by width-n ReLU networks on R^n in L1, indicating a phase transition.
• There exist width- O(k^2) depth-3 networks that cannot be approximated by width-O(k^1.5) depth-k networks, demonstrating a polynomial width efficiency lower bound (Theorem 4).
• Experiments show that narrow networks with size modestly larger than the polynomial lower bound can approximate wide shallow networks with high accuracy.
• Overall, the results provide evidence that depth may be more effective than width for the expressiveness of ReLU networks.