[论文解读] Numerically Recovering the Critical Points of a Deep Linear Autoencoder

Charles G. Frye, Neha S. Wadia+2 more|arXiv (Cornell University)|Jan 29, 2019

Model Reduction and Neural Networks参考文献 21被引用 5

一句话总结

本文评估了在深层线性自编码器中恢复临界点的数值方法，其中真实临界点可通过解析方法获得。研究发现，牛顿法优于梯度范数最小化方法，且严格的数值容差（例如 1e-10）对准确恢复至关重要；基于优化轨迹的采样方法会显著偏向低损失临界点。

ABSTRACT

Numerically locating the critical points of non-convex surfaces is a long-standing problem central to many fields. Recently, the loss surfaces of deep neural networks have been explored to gain insight into outstanding questions in optimization, generalization, and network architecture design. However, the degree to which recently-proposed methods for numerically recovering critical points actually do so has not been thoroughly evaluated. In this paper, we examine this issue in a case for which the ground truth is known: the deep linear autoencoder. We investigate two sub-problems associated with numerical critical point identification: first, because of large parameter counts, it is infeasible to find all of the critical points for contemporary neural networks, necessitating sampling approaches whose characteristics are poorly understood; second, the numerical tolerance for accurately identifying a critical point is unknown, and conservative tolerances are difficult to satisfy. We first identify connections between recently-proposed methods and well-understood methods in other fields, including chemical physics, economics, and algebraic geometry. We find that several methods work well at recovering certain information about loss surfaces, but fail to take an unbiased sample of critical points. Furthermore, numerical tolerance must be very strict to ensure that numerically-identified critical points have similar properties to true analytical critical points. We also identify a recently-published Newton method for optimization that outperforms previous methods as a critical point-finding algorithm. We expect our results will guide future attempts to numerically study critical points in large nonlinear neural networks.

研究动机与目标

• 评估数值方法在已知真实临界点的神经网络损失曲面中恢复临界点的保真度。
• 评估采样策略对临界点恢复偏差的影响。
• 确定识别真正临界点的合适数值容差。
• 比较梯度范数最小化、信赖域牛顿法与一种新型牛顿-MR 方法的性能。
• 理解现有方法在无偏采样临界点方面的局限性。

提出的方法

• 本研究采用具有已知解析临界点的深层线性自编码器作为基准，用于评估数值恢复方法。
• 测试了三种算法：梯度范数最小化（GNM）、信赖域牛顿法和一种新提出的牛顿-MR 方法。
• 采样策略包括在优化迭代中均匀采样以及在轨迹点中均匀采样，且在有无添加高斯噪声的情况下进行。
• 系统性地调整梯度范数的数值容差，发现 1e-10 的容差对准确恢复至关重要。
• 使用特征向量投影的熵来量化对主导方向的采样偏差。
• 将理论联系到化学物理、代数几何和经济学中的方法，以阐明算法选择的背景。

实验结果

研究问题

• RQ1 在已知真实值的前提下，数值方法能多准确地恢复深层线性自编码器的真实临界点？
• RQ2 识别与解析性质匹配的临界点的最优数值容差是什么？
• RQ3 基于优化轨迹的采样策略在多大程度上引入了临界点恢复的偏差？
• RQ4 梯度范数最小化（GNM）、信赖域牛顿法和牛顿-MR 方法在收敛速度和准确性方面如何比较？
• RQ5 为何向采样轨迹添加噪声无法完全消除偏差，为何更大的噪声反而可能增加对低损失临界点的偏差？

主要发现

• 牛顿-MR 方法在收敛速度和实际运行时间方面均优于梯度范数最小化和信赖域牛顿法。
• 梯度范数最小化方法经常陷入局部极小值，且收敛所需迭代次数多出两个数量级。
• 必须采用严格的数值容差 1e-10 才能准确恢复真实临界点的损失值和指标值，该容差高于以往常用的 1e-6。
• 在优化迭代中均匀采样引入了强烈偏差，使临界点更倾向于投影到主导特征向量上，表现为熵降低（3.05 比特 vs. 2.22 比特）。
• 添加高斯噪声并未完全消除偏差，且在某些情况下反而增加了对低损失临界点的偏差。
• 本研究证实，当前的采样方法并非无偏，且实现临界点的无偏采样仍是未解难题。