[Paper Review] Bounds on the Approximation Power of Feedforward Neural Networks

Mohammad Mehrabi, Aslan Tchamkerten+1 more|arXiv (Cornell University)|Jun 29, 2018

Neural Networks and Applications3 citations

TL;DR

This paper establishes tighter bounds on the approximation power of feedforward neural networks with piecewise linear activation functions, particularly ReLU. It derives improved lower bounds on network size (depth and width) required to approximate functions within a given error, and introduces a novel upper bound on the difference between two networks with identical weights but different activation functions, showing this difference grows exponentially with depth but is controlled by activation function similarity and network width.

ABSTRACT

The approximation power of general feedforward neural networks with piecewise linear activation functions is investigated. First, lower bounds on the size of a network are established in terms of the approximation error and network depth and width. These bounds improve upon state-of-the-art bounds for certain classes of functions, such as strongly convex functions. Second, an upper bound is established on the difference of two neural networks with identical weights but different activation functions.

Motivation & Objective

• To establish tighter lower bounds on the size (depth and width) of feedforward neural networks required to approximate a given function within a specified error.
• To improve upon existing state-of-the-art bounds for specific function classes, such as strongly convex functions, by analyzing network capacity via break point counting.
• To derive an upper bound on the difference between two networks with identical weights but different activation functions, quantifying the impact of activation function choice on network output.
• To analyze the tradeoff between network depth and width in terms of approximation power, showing depth provides exponential gains in representational capacity compared to width.

Proposed method

• Uses neuron state transition analysis from Raghu et al. (2017) to derive a tighter upper bound on the number of linear pieces (break points) a network can produce.
• Establishes a multiplicative factor $ d^d $ improvement over prior work (Yarotsky, 2017) in bounding the number of break points, where $ d $ is network depth.
• Combines the improved break point bound with error and dimension-dependent lower bounds to derive necessary conditions on depth, width, error, and input dimension for approximation.
• Introduces a recursive analysis of hidden unit output differences between two networks with identical weights but different activation functions $ ho $-Lipschitz and $ ho $-close in $ L^ atural $-norm.
• Derives a recurrence relation $ \varepsilon_{i+1}(\boldsymbol{x}) \leq \varepsilon + \delta A \sum_{j=1}^i \omega_j \varepsilon_j(\boldsymbol{x}) $ to bound the maximum output difference per layer.
• Uses the recurrence to derive the final bound: $ |f_1(\boldsymbol{x}) - f_2(\boldsymbol{x})| \leq \frac{||\sigma_1 - \sigma_2||_\infty}{\delta} \left( (\delta A \omega_f + 1)^{d_f} - 1 \right) $, showing exponential dependence on depth.

Experimental results

Research questions

• RQ1 How does network depth and width jointly constrain the approximation error for piecewise linear feedforward networks?
• RQ2 Can tighter lower bounds on network size be derived for specific function classes, such as strongly convex functions, using break point counting?
• RQ3 What is the maximum difference between two networks with identical weights but different activation functions, and how does it scale with depth and width?
• RQ4 How does the choice of activation function affect the output difference of otherwise identical networks, and can this be bounded independently of network size?
• RQ5 What is the role of activation function smoothness and Lipschitz constant in controlling the deviation between two networks with different activation functions?

Key findings

• The paper improves the upper bound on the number of linear pieces (break points) a ReLU network can produce by a factor of $ d^d $ compared to Yarotsky (2017), significantly tightening the theoretical understanding of network capacity.
• For strongly convex functions, the derived lower bounds on network size (depth and width) are tighter than previous state-of-the-art results, indicating that deeper networks are more efficient for such functions.
• An upper bound is established on the difference between two networks with identical weights but different activation functions: $ |f_1(\boldsymbol{x}) - f_2(\boldsymbol{x})| \leq \frac{||\sigma_1 - \sigma_2||_\infty}{\delta} \left( (\delta A \omega_f + 1)^{d_f} - 1 \right) $, showing exponential dependence on depth.
• The bound on output difference grows exponentially with depth $ d_f $, but is controlled by the similarity of activation functions (measured by $ ||\sigma_1 - \sigma_2||_\infty $) and the network's width $ \omega_f $, indicating that small changes in activation can lead to large output differences in deep networks.
• The analysis reveals that depth provides exponential gains in representational capacity compared to width, as depth leads to exponential growth in the number of linear pieces, while width only yields polynomial growth.
• The derived bounds are shown to be tight for certain classes of functions, such as those with high curvature, where the lower bounds on required network size match the upper bounds derived from break point counting.