[Paper Review] Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimation

Luca Pesce, Florent Krząkała+2 more|arXiv (Cornell University)|Feb 17, 2023

Bayesian Methods and Mixture Models4 citations

TL;DR

This paper provides exact asymptotic expressions for training and test errors in high-dimensional generalized linear models with Gaussian mixture data, revealing that Gaussian universality—where Gaussian data accurately predict performance—depends critically on the alignment between the target weights and cluster structure. The key finding is that universality breaks when there is correlation between the target and cluster means, even in homoscedastic mixtures, challenging the assumption that Gaussian data universally model real-world learning errors.

ABSTRACT

In this manuscript we consider the problem of generalized linear estimation on Gaussian mixture data with labels given by a single-index model. Our first result is a sharp asymptotic expression for the test and training errors in the high-dimensional regime. Motivated by the recent stream of results on the Gaussian universality of the test and training errors in generalized linear estimation, we ask ourselves the question: "when is a single Gaussian enough to characterize the error?". Our formula allow us to give sharp answers to this question, both in the positive and negative directions. More precisely, we show that the sufficient conditions for Gaussian universality (or lack of thereof) crucially depend on the alignment between the target weights and the means and covariances of the mixture clusters, which we precisely quantify. In the particular case of least-squares interpolation, we prove a strong universality property of the training error, and show it follows a simple, closed-form expression. Finally, we apply our results to real datasets, clarifying some recent discussion in the literature about Gaussian universality of the errors in this context.

Motivation & Objective

• To derive exact asymptotic expressions for training and test errors in generalized linear models under high-dimensional Gaussian mixture data.
• To investigate the conditions under which Gaussian universality holds—i.e., when errors on Gaussian mixtures match those on single-Gaussian data.
• To clarify the role of data structure (cluster means and covariances) versus task structure (target weights) in determining generalization performance.
• To validate theoretical findings on real datasets using random feature maps and synthetic regression tasks.
• To resolve conflicting literature claims about Gaussian universality in real-world learning scenarios.

Proposed method

• Employs the replica method from statistical physics to derive exact asymptotic expressions for errors in the proportional high-dimensional limit (n,d → ∞, α = n/d fixed).
• Analyzes generalized linear models with convex loss functions under a single-index model for labels, assuming data from a Gaussian mixture with arbitrary means and covariances.
• Derives sufficient conditions for universality: when target weights are isotropically distributed or aligned with low-dimensional signal subspaces.
• Introduces a theoretical framework to compute training and generalization errors via saddle-point equations for overlaps (ρ, π) and order parameters (q, h).
• Applies the theory to ridge regression and least-squares interpolation, showing closed-form expressions for training error in the latter.
• Validates results on real datasets using random feature maps and synthetic labels, comparing predictions from Gaussian and Gaussian mixture models.

Figure 1: An illustration of Gaussian universality with vanishing regularization $\lambda=0^{+}$ for a selection of datasets (MNIST, Fashion-MNIST, Cifar10) with a random teacher function, after a random feature map: Generalization (left) and training (right) errors as a function of the number of sa

Experimental results

Research questions

• RQ1 Under what conditions do training and test errors on Gaussian mixture data match those on single-Gaussian data?
• RQ2 How does the correlation between target weights and cluster means affect the universality of generalization error?
• RQ3 Can the training error in least-squares interpolation on Gaussian mixtures be expressed in a simple, closed-form expression?
• RQ4 To what extent does heteroscedasticity break Gaussian universality in high-dimensional generalized linear estimation?
• RQ5 Does the linear separability transition in Gaussian mixtures exhibit universal behavior, and if so, under what conditions?

Key findings

• The asymptotic training and test errors for generalized linear models on Gaussian mixture data are exactly characterized via a replica-based derivation, valid in the high-dimensional limit.
• Gaussian universality holds when the target weight vector is isotropically distributed on the sphere S^{d-1}, regardless of cluster structure.
• In ridge regression, the training error is universal and reduces to that of a single Gaussian with identity covariance, independent of cluster means and covariances.
• Universality breaks when the target weight correlates with cluster means—even in homoscedastic mixtures—due to misalignment between task and data structure.
• For least-squares interpolation, the training error follows a simple closed-form expression, confirming a strong universality property.
• Numerical validation on real datasets shows that Gaussian predictions closely match GMM performance when the teacher is uncorrelated with cluster structure, but diverge when correlation is present.

Figure 2: An illustration of Gaussian universality with finite regularization $\lambda$ for a selection of datasets (MNIST, Fashion-MNIST, Cifar10) with a random teacher function, after a random feature map: Generalization (left) and training (right) errors as a function of the number of samples per