[Paper Review] A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning

Ben Adlam, Jake Levinson+1 more|arXiv (Cornell University)|Dec 2, 2019

Random Matrices and ApplicationsMathematics42 references17 citations

TL;DR

This paper presents a random matrix theory framework to analyze random feature regression with mixtures of nonlinearities in high-dimensional settings, where both data size and model width grow large. It derives exact asymptotic formulas for training and test errors, showing that a learned mixture of nonlinearities can outperform the best single nonlinearity in both memorization capacity and generalization, offering new insights for kernel methods and neural architecture design.

ABSTRACT

One of the distinguishing characteristics of modern deep learning systems is that they typically employ neural network architectures that utilize enormous numbers of parameters, often in the millions and sometimes even in the billions. While this paradigm has inspired significant research on the properties of large networks, relatively little work has been devoted to the fact that these networks are often used to model large complex datasets, which may themselves contain millions or even billions of constraints. In this work, we focus on this high-dimensional regime in which both the dataset size and the number of features tend to infinity. We analyze the performance of random feature regression with features $F=f(WX+B)$ for a random weight matrix $W$ and random bias vector $B$, obtaining exact formulae for the asymptotic training and test errors for data generated by a linear teacher model. The role of the bias can be understood as parameterizing a distribution over activation functions, and our analysis directly generalizes to such distributions, even those not expressible with a traditional additive bias. Intriguingly, we find that a mixture of nonlinearities can improve both the training and test errors over the best single nonlinearity, suggesting that mixtures of nonlinearities might be useful for approximate kernel methods or neural network architecture design.

Motivation & Objective

• To understand the performance of deep learning models in the high-dimensional regime where both dataset size and model width tend to infinity.
• To analyze random feature regression with nonlinearities induced by random weights and bias distributions.
• To investigate whether mixtures of nonlinearities can improve generalization and memorization beyond the best single nonlinearity.
• To derive exact asymptotic expressions for training and test errors under a linear teacher model.
• To extend spectral analysis of kernel matrices to non-Gaussian data and non-zero bias distributions.

Proposed method

• Uses random matrix theory to analyze the resolvent of the kernel matrix $ F^ op F $, where $ F = f(WX + B) $, with random $ W $ and $ B $.
• Introduces a surrogate linearization $ F^{ ext{lin}} $ that preserves the spectral properties of $ F $, enabling tractable analysis.
• Derives exact formulas for the asymptotic training error in a noisy autoencoding task and test error under a linear teacher model.
• Applies the resolvent method to characterize the eigenvalue density of the kernel matrix, generalizing prior results to non-Gaussian and biased settings.
• Uses operator-valued free probability to compute test error, linking results to generalized cross-validation (GCV) metrics.
• Validates theoretical predictions via simulations on activation functions like leaky ReLU with varying slopes, showing high agreement with theory.

Experimental results

Research questions

• RQ1 Can a mixture of nonlinearities outperform the best single nonlinearity in terms of training and test error in high-dimensional random feature models?
• RQ2 How do bias distributions parameterize a family of activation functions, and how does this affect generalization and memorization?
• RQ3 What is the exact asymptotic behavior of the training and test errors when both data size and model width grow large?
• RQ4 How does the spectral density of the kernel matrix $ F^ op F $ depend on the input data distribution and the nonlinearity used?
• RQ5 Is there a deeper connection between the derived test error and generalized cross-validation (GCV) metrics in random feature models?

Key findings

• A mixture of nonlinearities achieves lower test error and better memorization capacity than the best single nonlinearity, even when the latter is optimally chosen.
• The theoretical prediction for test error asymptotically matches the generalized cross-validation (GCV) metric, suggesting a fundamental link between GCV and random feature learning.
• The eigenvalue density of the kernel matrix $ F^ op F $ is analytically characterized, extending prior work to non-Gaussian data and non-zero bias distributions.
• The surrogate linearization $ F^{ ext{lin}} $ preserves the spectral information of $ F $, enabling exact computation of error formulas.
• Simulations show excellent agreement between theoretical predictions and empirical results, even with a single sample, validating the asymptotic formulas.
• The performance gap between the optimal single nonlinearity and the optimal mixture is numerically detectable, confirming that mixtures provide a genuine advantage.