[Paper Review] The empirical size of trained neural networks

Kai Chen, Anthony Gamst+1 more|arXiv (Cornell University)|Nov 29, 2016

Neural Networks and Applications4 references3 citations

TL;DR

This paper empirically demonstrates that trained ReLU neural networks are vastly simpler than their parameter count suggests, exhibiting O(NK) linear pieces instead of the theoretically possible O(N^K). Through extensive experiments, it shows that standard initialization and training induce degenerate, sparse networks that remain simple throughout optimization, which explains their generalization success and enables linear training dynamics where low-frequency components are learned before high-frequency noise.

ABSTRACT

ReLU neural networks define piecewise linear functions of their inputs. However, initializing and training a neural network is very different from fitting a linear spline. In this paper, we expand empirically upon previous theoretical work to demonstrate features of trained neural networks. Standard network initialization and training produce networks vastly simpler than a naive parameter count would suggest and can impart odd features to the trained network. However, we also show the forced simplicity is beneficial and, indeed, critical for the wide success of these networks.

Motivation & Objective

• To empirically validate the theoretical claim that trained ReLU networks have far fewer linear pieces than their parameter count implies.
• To investigate how standard initialization and training procedures lead to network simplicity and degeneracy (e.g., zero neurons).
• To examine the impact of training hyperparameters—especially batch size—on network quality and optimization behavior.
• To explore why this inherent simplicity is beneficial for generalization and efficient learning.
• To demonstrate that training dynamics favor low-frequency components, enabling efficient learning of complex functions by prioritizing coarse features.

Proposed method

• Used standard feedforward ReLU networks with Glorot uniform initialization and Adadelta optimizer in Keras.
• Measured network size via sum of squared outputs on Gaussian noise, a proxy for Gaussian complexity.
• Trained networks for up to 50,000 epochs with single gradient steps per epoch, tracking mean squared error and size over time.
• Varied batch sizes (including remainder batches) to assess their impact on training quality and convergence.
• Trained on synthetic data with known frequency components (e.g., linear spline + noise) to isolate learning dynamics.
• Computed the difference between models trained on noisy and noiseless data to approximate noise fitting behavior.

Experimental results

Research questions

• RQ1 To what extent do standard initialization and training reduce the effective complexity of ReLU networks below theoretical maximums?
• RQ2 How do batch size and remainder batches influence the quality and stability of trained networks?
• RQ3 Why do networks trained on noisy data learn low-frequency components before high-frequency noise, and how does this affect generalization?
• RQ4 Can the observed simplicity and linear training dynamics be leveraged to improve model fitting, especially in noisy settings?
• RQ5 To what extent do degenerate neurons (identically zero functions) emerge during training, and how do they affect network behavior?

Key findings

• In 3-input, 20-layer networks, 30% of neurons in the final hidden layer were identically zero after only 5 training steps, confirming degeneracy beyond one-dimensional cases.
• Even with over 2500 data points, networks with 3 inputs and 4 hidden layers achieved a maximum sum of squared outputs of only 600, far below the 2500 expected for a perfect fit, indicating strong inductive bias toward simplicity.
• Batching with remainder groups significantly affected training outcomes, with smaller final batches introducing instability and altering convergence patterns.
• Networks trained on high-frequency sawtooth waves with zero-initialized biases exhibited trapped local minima, failing to improve beyond 8000–20,000 epochs despite sufficient capacity.
• Training on a noisy function (spline + noise) resulted in learning the low-frequency component first, with noise fitting occurring at the same rate as when trained on noise alone, confirming linear training dynamics.
• The difference between models trained on noisy and noiseless data closely approximated the noise, suggesting that 'artificial boosting'—adding a known component before training—could enhance noise fitting in practice.