[Paper Review] Achieve the Minimum Width of Neural Networks for Universal Approximation

Yongqiang Cai|arXiv (Cornell University)|Sep 23, 2022

Model Reduction and Neural Networks4 citations

TL;DR

This paper establishes a universal lower bound of $ w^{*}_{ ext{min}} = \max(d_x, d_y) $ for the minimum width of feed-forward neural networks required for universal approximation in both $ L^p $ and continuous norms on compact domains. It proves that this bound is tight by constructing networks using leaky-ReLU and UOE activation functions, leveraging neural ODEs and flow map approximation to achieve optimal width, resolving a long-standing open question on minimal width for arbitrary activation functions.

ABSTRACT

The universal approximation property (UAP) of neural networks is fundamental for deep learning, and it is well known that wide neural networks are universal approximators of continuous functions within both the $L^p$ norm and the continuous/uniform norm. However, the exact minimum width, $w_{\min}$, for the UAP has not been studied thoroughly. Recently, using a decoder-memorizer-encoder scheme, \citet{Park2021Minimum} found that $w_{\min} = \max(d_x+1,d_y)$ for both the $L^p$-UAP of ReLU networks and the $C$-UAP of ReLU+STEP networks, where $d_x,d_y$ are the input and output dimensions, respectively. In this paper, we consider neural networks with an arbitrary set of activation functions. We prove that both $C$-UAP and $L^p$-UAP for functions on compact domains share a universal lower bound of the minimal width; that is, $w^*_{\min} = \max(d_x,d_y)$. In particular, the critical width, $w^*_{\min}$, for $L^p$-UAP can be achieved by leaky-ReLU networks, provided that the input or output dimension is larger than one. Our construction is based on the approximation power of neural ordinary differential equations and the ability to approximate flow maps by neural networks. The nonmonotone or discontinuous activation functions case and the one-dimensional case are also discussed.

Motivation & Objective

• To determine the exact minimum width required for universal approximation in feed-forward neural networks with arbitrary activation functions.
• To establish a universal lower bound on the minimal width that applies to both $ L^p $- and continuous-norm universal approximation.
• To construct neural network architectures that achieve this minimal width, demonstrating tightness of the bound.
• To explore the role of activation functions—particularly non-monotone and discontinuous ones—in achieving minimal width.
• To connect universal approximation to topological properties via neural ODEs and flow map approximation.

Proposed method

• Derives a universal lower bound $ w^{*}_{\text{min}} = \max(d_x, d_y) $ using topological arguments and counterexamples based on level sets and homeomorphisms.
• Proposes a novel construction based on neural ordinary differential equations (ODEs), showing that flow maps of ODEs can be approximated by neural networks to achieve universal approximation at minimal width.
• Uses the encoder-memorizer-decoder framework for $ C $-UAP, incorporating discontinuous activation functions like $ \lfloor x \rfloor $ and the UOE (universal ordering of extrema) function.
• Demonstrates that leaky-ReLU networks achieve the minimal width $ \max(d_x, d_y, 2) $, and leaky-ReLU+ABS or UOE+FLOOR networks achieve $ \max(d_x, d_y) $, proving optimality under certain conditions.
• Applies topological reasoning to show that continuous, monotone activations (e.g., leaky-ReLU) cannot achieve $ w^{*}_{\text{min}} $ when $ d_x \geq d_y $, due to homeomorphism constraints.
• Analyzes the case of $ d_x = 1 $ using UOE activation to achieve $ w^{*}_{\text{min}} = d_y $, extending to higher dimensions via topological invariance.

Experimental results

Research questions

• RQ1 What is the universal lower bound on the minimal width required for universal approximation in feed-forward neural networks with arbitrary activation functions?
• RQ2 Can this lower bound be achieved by specific activation functions, and if so, which ones?
• RQ3 How do neural ODEs and flow map approximation contribute to achieving minimal-width universal approximation?
• RQ4 Why do continuous, monotone activation functions fail to achieve the minimal width in certain cases, and what topological constraints underlie this?
• RQ5 Can the minimal width $ \max(d_x, d_y) $ be achieved using only continuous activation functions, or are discontinuous or non-monotone functions necessary?

Key findings

• The universal lower bound for the minimal width of neural networks with arbitrary activation functions is $ w^{*}_{\text{min}} = \max(d_x, d_y) $, valid for both $ L^p $- and continuous-norm universal approximation on compact domains.
• This bound is tight: leaky-ReLU networks achieve $ w^{*}_{\text{min}} = \max(d_x, d_y, 2) $, and leaky-ReLU+ABS or UOE+FLOOR networks achieve $ w^{*}_{\text{min}} = \max(d_x, d_y) $, proving optimality.
• For $ L^p $-UAP, the minimal width $ \max(d_x, d_y) $ is achieved by leaky-ReLU networks when $ d_x > 1 $ or $ d_y > 1 $, using the approximation power of neural ODEs and flow maps.
• For $ C $-UAP, the minimal width $ \max(d_x, d_y) $ is achieved using UOE+FLOOR networks, particularly in the case $ d_x = 1 $, where $ w^{*}_{\text{min}} = d_y $.
• Topological constraints show that continuous, monotone activations (e.g., leaky-ReLU) cannot achieve $ w^{*}_{\text{min}} $ when $ d_x \geq d_y $, due to the inability of homeomorphisms to approximate functions with non-trivial level sets.
• The construction via neural ODEs provides a new geometric and topological perspective on universal approximation, linking it to the approximation of diffeomorphisms and flow maps.