[Paper Review] On the Spectral Bias of Neural Networks

Nasim Rahaman, Aristide Baratin+6 more|arXiv (Cornell University)|Jun 22, 2018

Neural Networks and Applications164 citations

TL;DR

The paper shows that deep ReLU networks exhibit a spectral bias, learning low-frequency (smooth) components first, and analyzes how data-manifold geometry affects learning of higher frequencies.

ABSTRACT

Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with $100\%$ accuracy. In this work, we present properties of neural networks that complement this aspect of expressivity. By using tools from Fourier analysis, we show that deep ReLU networks are biased towards low frequency functions, meaning that they cannot have local fluctuations without affecting their global behavior. Intuitively, this property is in line with the observation that over-parameterized networks find simple patterns that generalize across data samples. We also investigate how the shape of the data manifold affects expressivity by showing evidence that learning high frequencies gets \emph{easier} with increasing manifold complexity, and present a theoretical understanding of this behavior. Finally, we study the robustness of the frequency components with respect to parameter perturbation, to develop the intuition that the parameters must be finely tuned to express high frequency functions.

Motivation & Objective

• Motivate and formalize the spectral bias of neural networks using Fourier analysis.
• Characterize the Fourier spectrum of ReLU networks and its decay properties.
• Demonstrate that lower frequencies are learned faster and are more robust to parameter perturbations.
• Investigate how data-manifold geometry affects the learnability of higher frequencies.
• Provide empirical and theoretical insights on training dynamics and manifold effects.

Proposed method

• Represent ReLU networks as continuous piecewise-linear (CPWL) functions and express f(x) as a sum over linear regions (Eq. 2).
• Compute the Fourier transform of ReLU networks and derive its form as a rational function (Eq. 6).
• Show that in most directions the spectrum decays as k^{-(d+1)} while certain directions decay as slow as k^{-2}.
• Relate spectrum bounds to Lipschitz constants and network parameters (Eq. 7).
• Conduct synthetic experiments to observe spectral bias and robustness to parameter perturbations (Experiments 1 and 2).
• Study real-data behavior on MNIST via frequency-noise experiments (Experiment 3) and generalized frequency projections (Experiment 4).
• Examine the effect of data-manifold geometry on high-frequency learnability (Section 4 and Experiments 5-6).

Experimental results

Research questions

• RQ1 Do deep ReLU networks exhibit a systematic bias toward learning low-frequency components?
• RQ2 How does the Fourier spectrum of a trained network relate to its architecture and parameters?
• RQ3 Does the data manifold shape influence the ease of learning higher-frequency components?
• RQ4 How robust are low- vs high-frequency components to perturbations in network parameters?
• RQ5 How does embedding data on complex manifolds affect frequency-based learnability in practice?

Key findings

• Networks prioritize learning lower frequencies before higher ones during training.
• Lower-frequency components are more robust to random perturbations of network parameters than higher-frequency components.
• The Fourier spectrum of ReLU networks decays anisotropically, with k^{-(d+1)} decay in most directions and as slow as k^{-2} in directions orthogonal to region faces.
• Higher frequencies can become easier to learn when data lies on more complex, lower-dimensional manifolds.
• The manifold embedding can enable large input-space frequencies to be represented by smaller network frequencies, depending on manifold geometry.
• Empirical results on MNIST show frequency-dependent effects consistent with synthetic experiments.