[Paper Review] The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies

Ronen Basri, David Jacobs+2 more|arXiv (Cornell University)|Jun 2, 2019

Neural Networks and ApplicationsComputer Science26 references90 citations

TL;DR

The paper analyzes how gradient descent-trained neural networks learn functions of varying frequency, showing low-frequency components are learned faster and highlighting the impact of bias on learning odd frequencies.

ABSTRACT

We study the relationship between the frequency of a function and the speed at which a neural network learns it. We build on recent results that show that the dynamics of overparameterized neural networks trained with gradient descent can be well approximated by a linear system. When normalized training data is uniformly distributed on a hypersphere, the eigenfunctions of this linear system are spherical harmonic functions. We derive the corresponding eigenvalues for each frequency after introducing a bias term in the model. This bias term had been omitted from the linear network model without significantly affecting previous theoretical results. However, we show theoretically and experimentally that a shallow neural network without bias cannot represent or learn simple, low frequency functions with odd frequencies. Our results lead to specific predictions of the time it will take a network to learn functions of varying frequency. These predictions match the empirical behavior of both shallow and deep networks.

Motivation & Objective

• Motivate and analyze why overparameterized networks generalize well by examining frequency-based learning dynamics.
• Characterize how training data on a hypersphere leads to spherical harmonic eigenfunctions governing learning speeds.
• Show how a bias term affects the learnability of odd-frequency components and the resulting convergence behavior.
• Provide theoretical predictions for learning times per frequency and validate them experimentally on shallow and deep networks.

Proposed method

• Model the gradient descent dynamics of a two-layer network in a linearized regime with ReLU activations.
• Define the Z matrix and the Gram/H infinity matrices to capture training dynamics.
• Derive eigenvalues/eigenfunctions of H^infty under uniform spherical data, showing spherical harmonics as eigenfunctions.
• Extend the model to include bias and show how it changes the eigenstructure and learnability of odd frequencies.
• Use the Funk-Hecke theorem to analyze convolution kernels on the sphere and obtain closed-form eigenvalues for K^infty and Kbar^infty.
• Empirically validate convergence rates for different frequencies and network depths against the quadratic k-scaling prediction.

Experimental results

Research questions

• RQ1 How does the frequency of a target function affect the convergence rate of gradient descent in overparameterized networks?
• RQ2 What is the impact of including a bias term on the learnability of odd-frequency components?
• RQ3 Do the theoretical eigenvalues/eigenfunctions translate to observed learning times in shallow and deep networks?
• RQ4 How do results extend from one-dimensional circular data to higher-dimensional hyperspheres?
• RQ5 Can the observed frequency-based learning dynamics explain generalization and early stopping phenomena?

Key findings

• Low-frequency components of target functions are learned faster than high-frequency components under gradient descent.
• For bias-free networks, odd frequencies with k ≥ 3 lie in the null space and cannot be learned or represented.
• With bias, odd frequencies are learnable and the eigenvectors remain spherical harmonics, yielding comparable learning rates across frequencies.
• Convergence time for frequency k scales quadratically with k (e.g., k^2 in 1D) and grows roughly as k^d in dimension d, aligning with experimental results across shallow and deep architectures.
• Empirical convergence times match the theoretical predictions across two-layer nets, deep nets, and networks with skip connections, with bias improving odd-frequency learnability.
• The analysis suggests gradient descent acts as a frequency-based regularizer, favoring low-frequency (smoother) solutions during training.