[Paper Review] Approximating Continuous Functions by ReLU Nets of Minimal Width

Boris Hanin, Mark Sellke|arXiv (Cornell University)|Oct 31, 2017

Neural Networks and Applications18 references143 citations

TL;DR

The paper determines the minimal hidden-layer width required for ReLU nets to approximate any continuous function of d_in variables, showing the threshold is d_in+1 and providing an upper bound of d_in+d_out.

ABSTRACT

This article concerns the expressive power of depth in deep feed-forward neural nets with ReLU activations. Specifically, we answer the following question: for a fixed $d_{in}\geq 1,$ what is the minimal width $w$ so that neural nets with ReLU activations, input dimension $d_{in}$, hidden layer widths at most $w,$ and arbitrary depth can approximate any continuous, real-valued function of $d_{in}$ variables arbitrarily well? It turns out that this minimal width is exactly equal to $d_{in}+1.$ That is, if all the hidden layer widths are bounded by $d_{in}$, then even in the infinite depth limit, ReLU nets can only express a very limited class of functions, and, on the other hand, any continuous function on the $d_{in}$-dimensional unit cube can be approximated to arbitrary precision by ReLU nets in which all hidden layers have width exactly $d_{in}+1.$ Our construction in fact shows that any continuous function $f:[0,1]^{d_{in}} o\mathbb R^{d_{out}}$ can be approximated by a net of width $d_{in}+d_{out}$. We obtain quantitative depth estimates for such an approximation in terms of the modulus of continuity of $f$.

Motivation & Objective

• Determine the minimal hidden-layer width w_min(d_in, d_out) for ReLU nets to approximate any continuous function f:[0,1]^{d_in} -> R^{d_out}
• Show that if hidden-layer widths are at most d_in, nets have limited expressivity regardless of depth
• Provide an upper bound construction achieving width d_in+d_out that approximates any continuous function
• Quantify depth in terms of the modulus of continuity of f for the width-d_in+d_out construction
• Establish a matching lower bound demonstrating width at least d_in+1 is necessary for universal approximation

Proposed method

• Introduce the w_min(d_in,d_out) threshold for ReLU nets without skip connections
• Prove an upper bound w_min(d_in,d_out) ≤ d_in+d_out via a max-min string construction that replicates any continuous function on a compact set
• Show any continuous f can be approximated on a compact K by a ReLU net of width d_in+d_out with depth depending on the modulus of continuity ω_f
• Utilize Propositions: (i) max-min string realization by ReLU nets of width d_in+d_out (Proposition 2) and (ii) approximation by max-min strings with controlled length L = (O(diam(K))/ω_f^{-1}(ε))^{d_in+1} (Proposition 3)
• Develop a geometric corner-cutting argument (Lemma 5) to extend ε-approximations to larger domains and derive the depth bound
• Demonstrate a lower bound by constructing a function with a level-set geometry that cannot be captured by width-d_in networks, ensuring w_min(d_in,·) ≥ d_in+1

Experimental results

Research questions

• RQ1 What is the minimal hidden-layer width w_min(d_in, d_out) that enables ε-approximation of every continuous f:[0,1]^{d_in} → R^{d_out}?
• RQ2 Is d_in+1 the sharp lower bound for universal approximation with ReLU nets of fixed width in hidden layers?
• RQ3 Can every continuous function on [0,1]^{d_in} be approximated by a ReLU net of width exactly d_in+d_out, and what is the required depth?
• RQ4 How does the modulus of continuity of f influence the depth required for approximation when width is limited to d_in+d_out?
• RQ5 What obstruction prevents width less than d_in+1 from achieving universal approximation without skip connections?

Key findings

• The minimal width for universal approximation with ReLU nets is exactly d_in+1 (lower bound) and can be achieved with width up to d_in+d_out (upper bound)
• Any continuous function f:[0,1]^{d_in}→R^{d_out} can be ε-approximated by a ReLU net with hidden widths bounded by d_in+d_out, with depth depending on the modulus of continuity ω_f
• The upper bound construction relies on max-min string representations and a depth bound that scales as (diam(K)/ω_f^{-1}(ε))^{d_in+1}
• A density argument shows that width d_in is insufficient for universal approximation, yielding a positive η such that no width-d_in network can approximate all continuous functions on [0,1]^{d_in}
• Skip connections are not allowed in their setup; allowing them would trivialize the width bound by enabling networks with width 1 to approximate any continuous function with sufficient depth
• The lower bound is established by constructing a function whose level-set geometry prevents width-d_in networks from universal approximation; this uses structural properties of ReLU nets and convex/affine pieces