[论文解读] Dimensionality compression and expansion in Deep Neural Networks

Stefano Recanatesi, Matthew Farrell+4 more|arXiv (Cornell University)|Jun 2, 2019

Generative Adversarial Networks and Image Synthesis参考文献 47被引用 42

一句话总结

本文使用内在维度估计来表明深度网络在两个阶段产生低维表示流形：在早期层扩张，在后期层压缩，SGD正则化平衡这些力量以帮助泛化。

ABSTRACT

Datasets such as images, text, or movies are embedded in high-dimensional spaces. However, in important cases such as images of objects, the statistical structure in the data constrains samples to a manifold of dramatically lower dimensionality. Learning to identify and extract task-relevant variables from this embedded manifold is crucial when dealing with high-dimensional problems. We find that neural networks are often very effective at solving this task and investigate why. To this end, we apply state-of-the-art techniques for intrinsic dimensionality estimation to show that neural networks learn low-dimensional manifolds in two phases: first, dimensionality expansion driven by feature generation in initial layers, and second, dimensionality compression driven by the selection of task-relevant features in later layers. We model noise generated by Stochastic Gradient Descent and show how this noise balances the dimensionality of neural representations by inducing an effective regularization term in the loss. We highlight the important relationship between low-dimensional compressed representations and generalization properties of the network. Our work contributes by shedding light on the success of deep neural networks in disentangling data in high-dimensional space while achieving good generalization. Furthermore, it invites new learning strategies focused on optimizing measurable geometric properties of learned representations, beginning with their intrinsic dimensionality.

研究动机与目标

• 研究为何深度神经网络能够有效解决高维分类任务。
• 量化数据及学习表示在跨越网络层的内在维度。
• 理解训练动态如何影响维度与泛化。

提出的方法

• 应用最先进的内在维度估计技术来测量局部和全局流形维度。
• 训练两个网络（在 Fashion-MNIST 上的 DeepNet 和在 CIFAR-10/CIFAR-100 上的 ResNet）并分析逐层维度。
• 比较训练前/训练后的维度以识别扩张与压缩阶段。
• 将 SGD 模型为产生一种有效正则化项，惩罚表示的维度。
• 分析层类型（卷积、全连接）和非线性（ReLU）对维度的影响。
• 使用线性和非线性分析来解释维度如何与任务需求和特征选择相关。

实验结果

研究问题

• RQ1 在深度网络的各层中，数据流形及学习表示的内在维度是多少？
• RQ2 神经网络在学习过程中是否呈现出明显的维度扩张与压缩阶段？
• RQ3 随机梯度下降如何通过一种有效的正则化影响表示的维度？
• RQ4 维度与深度结构的泛化性及任务性能之间的关系如何？

主要发现

• 深度网络中的表示流形相对于层大小而言非常低维。
• 维度在早期层扩张，在后期层压缩。
• ReLU 非线性增加维度，而位于 ReLU 之前的权重矩阵驱动表示压缩。
• SGD 引入一种有效的正则化，压缩与任务需求相符的方向，同时抑制与任务无关的方向，平衡扩张。
• 更宽的网络不一定增加学习到的流形维度，表明维度受任务需求和 SGD 而非网络规模控制。
• 维度降低与更好的泛化相关，并可用于指导网络结构规模和正则化策略。