[論文レビュー] Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval

Guillaume Braun, Han Bao+2 more|arXiv (Cornell University)|Jan 30, 2026

Advanced Electron Microscopy Techniques and Applications被引用数 0

ひとこと要約

tldr: Spectral Gradient Descent (SpecGD) mitigates variance-driven misalignment caused by anisotropic inputs in a phase retrieval model, achieving faster and more stable alignment than standard Gradient Descent (GD) through a three-dimensional invariant manifold analysis and sign-based updates. Theoretical results and empirical tests show SpecGD’s robustness under broader anisotropic covariances, including power-law spectra.

ABSTRACT

Spectral gradient methods, such as the Muon optimizer, modify gradient updates by preserving directional information while discarding scale, and have shown strong empirical performance in deep learning. We investigate the mechanisms underlying these gains through a dynamical analysis of a nonlinear phase retrieval model with anisotropic Gaussian inputs, equivalent to training a two-layer neural network with the quadratic activation and fixed second-layer weights. Focusing on a spiked covariance setting where the dominant variance direction is orthogonal to the signal, we show that gradient descent (GD) suffers from a variance-induced misalignment: during the early escaping stage, the high-variance but uninformative spike direction is multiplicatively amplified, degrading alignment with the true signal under strong anisotropy. In contrast, spectral gradient descent (SpecGD) removes this spike amplification effect, leading to stable alignment and accelerated noise contraction. Numerical experiments confirm the theory and show that these phenomena persist under broader anisotropic covariances.

研究の動機と目的

• Understand how anisotropic covariance affects learning dynamics in phase retrieval with a two-layer quadratic network.
• Characterize the learning dynamics under spectral gradient updates and compare with standard gradient descent.
• Identify mechanisms by which SpecGD avoids spike amplification and improves alignment.
• Extend insights to broader anisotropic covariances beyond the stylized spiked model.
• Provide theoretical and empirical validation of the proposed dynamics and benefits.

提案手法

• Model data as phase retrieval with Gaussian inputs having spiked covariance orthogonal to the signal.
• Analyze SpecGD where the gradient update is replaced by its polar factor to preserve directional information while discarding scale.
• Show a three-dimensional invariant manifold M spanned by the signal, spike, and bulk components and reduce dynamics to coefficients a, b, c.
• Derive continuous-time reduced ODEs for Gradient Flow (GF) and SpecGF, highlighting sign-based, scale-invariant updates for SpecGD.
• Compare discrete-time SpecGD and GD, including learning-rate regimes and stage-wise behavior, with proofs and barriers.
• Conduct numerical experiments including power-law covariance to validate robustness.

実験結果

リサーチクエスチョン

• RQ1 How does anisotropy in input covariances affect learning dynamics in phase retrieval under GD and SpecGD?
• RQ2 Can SpecGD prevent spike-driven misalignment and promote earlier, stable alignment with the target signal?
• RQ3 What are the Stage I (growth) and Stage II (alignment) dynamics under SpecGD versus GD?
• RQ4 Do the theoretical findings extend beyond the spiked covariance model to general anisotropic covariances such as power-law spectra?
• RQ5 How do continuous-time dynamics translate to discrete-time updates and learning-rate choices?

主な発見

• SpecGD induces a sign-based, scale-invariant update on an adaptive basis, preventing amplification of uninformative spike directions.
• Training dynamics collapse to a three-parameter invariant manifold with coordinated growth of signal, spike, and bulk components.
• Stage I under SpecGD shows uniform, quadratic growth of all coefficients at constant time, independent of dimension d, unlike GD.
• Stage II under SpecGD yields sustained growth of the signal coefficient and bounded growth of spike and bulk, leading to alignment (Align ≈ 1).
• GD experiences spike-dominated growth in Stage I and delayed signal growth in Stage II, resulting in longer overall training to achieve alignment.
• Numerical experiments confirm theory and show SpecGD robustness under power-law covariances and finite-sample settings.