[论文解读] Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction

Dominik Stöger, Mahdi Soltanolkotabi|Publication Server of the Catholic University Eichstätt-Ingolstadt (Catholic University of Eichstätt-Ingolstadt)|Jun 28, 2021

Sparse and Compressive Sensing Techniques参考文献 62被引用 13

一句话总结

该论文表明，在过参数化的低秩矩阵重构中，小的随机初始化会引发隐式的谱偏差，导致梯度下降沿类似于谱方法的轨迹行进。论文证明了三阶段收敛：（I）谱对齐，（II）鞍点避开，（III）局部精炼，进而在测量算子满足弱条件下实现全局最优性与强泛化保证。

ABSTRACT

Recently there has been significant theoretical progress on understanding the convergence and generalization of gradient-based methods on nonconvex losses with overparameterized models. Nevertheless, many aspects of optimization and generalization and in particular the critical role of small random initialization are not fully understood. In this paper, we take a step towards demystifying this role by proving that small random initialization followed by a few iterations of gradient descent behaves akin to popular spectral methods. We also show that this implicit spectral bias from small random initialization, which is provably more prominent for overparameterized models, also puts the gradient descent iterations on a particular trajectory towards solutions that are not only globally optimal but also generalize well. Concretely, we focus on the problem of reconstructing a low-rank matrix from a few measurements via a natural nonconvex formulation. In this setting, we show that the trajectory of the gradient descent iterations from small random initialization can be approximately decomposed into three phases: (I) a spectral or alignment phase where we show that that the iterates have an implicit spectral bias akin to spectral initialization allowing us to show that at the end of this phase the column space of the iterates and the underlying low-rank matrix are sufficiently aligned, (II) a saddle avoidance/refinement phase where we show that the trajectory of the gradient iterates moves away from certain degenerate saddle points, and (III) a local refinement phase where we show that after avoiding the saddles the iterates converge quickly to the underlying low-rank matrix. Underlying our analysis are insights for the analysis of overparameterized nonconvex optimization schemes that may have implications for computational problems beyond low-rank reconstruction.

研究动机与目标

• 理解在过参数化的非凸优化中，小的随机初始化所引入的隐式归纳偏置。
• 解释为何从较小随机初始化出发的梯度下降在模型过参数化的情况下仍能实现良好泛化。
• 形式化刻画梯度下降在低秩矩阵重构中的轨迹，超越标准的景观分析。
• 为低秩矩阵恢复的一个自然非凸公式建立可证明的收敛与泛化保证。
• 弥合过参数化设置下随机初始化的实际成功与理论理解之间的差距。

提出的方法

• 分析具有自然损失函数的非凸低秩矩阵重构问题中梯度下降的动力学。
• 识别出三阶段轨迹：（I）谱对齐，（II）鞍点避开，（III）局部精炼。
• 使用留一法分析与矩阵扰动理论，控制迭代点列空间与真实低秩矩阵对齐的演化。
• 证明小的随机初始化会引发类似于谱初始化的谱偏差，从而实现与底层矩阵的快速对齐。
• 通过证明迭代点避开退化鞍点并迅速向解收敛，从而实现全局最小值的收敛。
• 采用算子范数界与对测量算子的假设（例如，受限等距性质）来控制误差传播。

实验结果

研究问题

• RQ1 小的随机初始化如何影响过参数化低秩矩阵恢复中的优化轨迹？
• RQ2 为何从较小随机初始化出发的梯度下降在过参数化情况下仍能实现良好泛化？
• RQ3 小的随机初始化的隐式偏差能否被形式化地与谱方法关联？
• RQ4 在此非凸设置下，梯度下降动力学的显著阶段是什么？
• RQ5 在何种条件下算法能实现全局收敛并良好泛化？

主要发现

• 小的随机初始化会引发隐式谱偏差，使梯度下降迅速与真实低秩矩阵的列空间对齐。
• 优化轨迹可分解为三个阶段：谱对齐、鞍点避开与局部精炼，每个阶段均有可证明的保证。
• 由于梯度流的方向，具有严格负曲率的鞍点会被避开，从而确保收敛至全局最小值。
• 在对齐阶段之后，迭代点迅速收敛至真实低秩矩阵，误差以线性速率下降。
• 较小的初始化尺度可提升泛化性能，与深度学习中的经验观察一致。
• 理论界表明，列空间对齐误差的衰减速率与真实矩阵的最小奇异值成正比。