[论文解读] Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

Yiping Lu, Aoxiao Zhong+2 more|arXiv (Cornell University)|Oct 27, 2017

Model Reduction and Neural Networks参考文献 48被引用 153

一句话总结

本文将流行深度网络架构与常微分方程的数值离散化联系起来，提出一个线性多步 LM-架构并应用于 ResNet/ResNeXt，并展示在参数压缩潜力下的性能提升；同时将随机训练解读为随机动态系统。

ABSTRACT

In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters. In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress ($>50$\%) the original networks while maintaining a similar performance. This can be explained mathematically using the concept of modified equation from numerical analysis. Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture. As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

研究动机与目标

• 通过将架构与微分方程离散化联系起来，为深度网络设计原理提供动机。
• 引入 LM-架构并将其应用于 ResNet/ResNeXt，在更少参数的情况下提高准确率。
• 通过数值分析的修正方程来解释性能提升。
• 将随机训练视为近似随机动态系统以提升泛化能力。

提出的方法

• 将 ResNet/ResNeXt 及相关网络映射为求解 u_t = f(u) 的数值方案（前向欧拉、后向欧拉、龙格-库塔）。
• 提出 LM-架构：u_{n+1} = (1 - k_n) u_n + k_n u_{n-1} + f(u_n) with trainable k_n.
• 将 LM-架构应用于 ResNet/ResNeXt 以形成 LM-ResNet/LM-ResNeXt 并在 CIFAR 和 ImageNet 上进行评估。
• 分析修正方程以解释性能提升与稳定性。
• 将随机学习策略（噪声注入）描述为对随机动力学的近似，并将其扩展到 LM-架构（随机深度）。

实验结果

研究问题

• RQ1 深度网络架构是否可以被解释为微分方程的离散化，这一解释是否能指导架构设计？
• RQ2 在 CIFAR 和 ImageNet 上，LM-架构是否能提高 ResNet/ResNeXt 的性能和/或参数效率？
• RQ3 修正方程如何解释 LM-架构所带来的增益？
• RQ4 能否将随机训练策略理解为随机动态系统，并使 LM-架构受益？

主要发现

• LM-ResNet/LM-ResNeXt 在 CIFAR 和 ImageNet 上以相似参数量达到比对应 ResNet/ResNeXt 更高的准确性。
• 在 CIFAR 上，LM-ResNet/LM-ResNeXt 能在显著压缩原始网络的同时保持相似的性能（>50% 以上）。
• 在 CIFAR-10/ CIFAR-100 上，LM-ResNet/LM-ResNeXt 在不同深度下相对于基线架构显示出显著改进。
• 在 ImageNet 上，LM-ResNet50/ResNet50 与 LM-ResNet101/ResNet101 展示出在可比参数预算下的 top-1/top-5 准确率提升（例如 LM-ResNet50 top-1 23.8 对比 ResNet50 24.7；LM-ResNet101 top-1 22.6 对比 ResNet101 23.6）。
• 随机深度和其他噪声注入可以进一步提升性能，并且可以通过随机动态系统解释自然地并入 LM-架构。
• 修正方程分析解释了 LM-结构系数（k_n）如何影响学习动力学的加速和稳定性。