[论文解读] The role of regularization in classification of high-dimensional noisy Gaussian mixture

Francesca Mignacco, Florent Krząkała+2 more|arXiv (Cornell University)|Feb 26, 2020

Advanced Scientific Research Methods被引用 32

一句话总结

本文对高维两高斯混合在带噪声 regime 下，对正则化凸分类器（ridge, hinge, logistic）进行了严格的渐近分析，推导了泛化和训练误差的固定点公式，并与贝叶斯最优性能进行比较。

ABSTRACT

We consider a high-dimensional mixture of two Gaussians in the noisy regime where even an oracle knowing the centers of the clusters misclassifies a small but finite fraction of the points. We provide a rigorous analysis of the generalization error of regularized convex classifiers, including ridge, hinge and logistic regression, in the high-dimensional limit where the number $n$ of samples and their dimension $d$ go to infinity while their ratio is fixed to $\\alpha= n/d$. We discuss surprising effects of the regularization that in some cases allows to reach the Bayes-optimal performances. We also illustrate the interpolation peak at low regularization, and analyze the role of the respective sizes of the two clusters.

研究动机与目标

• Motivate the study of high-dimensional classification in Gaussian mixtures with noise and unknown centroid.
• Derive rigorous asymptotic formulas for generalization and training error under ridge, hinge, and logistic losses.
• Analyze how regularization strength and cluster sizes affect closeness to Bayes-optimal performance.
• Characterize the training loss landscape and separability transitions in the high-dimensional limit.

提出的方法

• Model data as a two-cluster Gaussian mixture with centroids and noise, and study regualrized empirical risk minimization with convex loss functions.
• Use Gordon’s minimax inequalities to transform the high-dimensional optimization into a tractable auxiliary problem.
• Derive fixed-point equations for overlap m, length q, and auxiliary variables (\u0013gamma, ￭hat m, ￭hat q, ￭hat gamma) that determine generalization/training quantities.
• Provide explicit expressions for the generalization error via Q-function and for the training loss in the d -> infinity limit.
• Analyze Bayes-optimal estimator and a plug-in Hebb-like estimator that can achieve Bayes-optimal performance in certain regimes.
• Discuss interpretations via replica theory and state evolution of AMP.

实验结果

研究问题

• RQ1 How does regularization (ridge, hinge, logistic) affect the generalization error in high-dimensional Gaussian mixture classification under noise?
• RQ2 What are the fixed-point relationships governing overlap with the true centroid and the norm of the classifier in the high-dimensional limit?
• RQ3 To what extent can regularized empirical risk minimization reach Bayes-optimal performance, and under which conditions?
• RQ4 How does cluster size asymmetry (rho != 0.5) influence separability, interpolation behavior, and optimal regularization?
• RQ5 What is the structure of the training loss landscape in high dimensions, and how does it relate to phase transitions in separability?

主要发现

• Rigorous closed-form asymptotic formulas are obtained for generalization and training error for any convex loss under regularization in the high-dimensional limit.
• The generalization error is given by a fixed-point system involving m, q, gamma, and b, with m and q expressed in terms of hat_m, hat_q, lambda, and hat_gamma.
• Bayes-optimal performance can be reached by certain plug-in estimators (e.g., Hebb-like weight) in some regimes, even though regularized ERM may not always achieve it.
• Regularization can improve performance and, in some symmetric cases, yield Bayes-optimal performance as lambda grows, while in non-symmetric cases optimal lambda remains finite.
• For linearly separable data, hinge and logistic losses converge to the same test error as regularization vanishes, illustrating implicit regularization and connections to double-descent phenomena.
• The analysis yields a phase-transition boundary for separability, with alpha* depending on cluster variance and rho; data become perfectly separable below this threshold, and MLE may not exist above it.
• numerical simulations at moderate dimensions (e.g., d=1000) corroborate the theoretical predictions.