[论文解读] RMNet: Equivalently Removing Residual Connection from Networks

Fanxu Meng, Hao Cheng+3 more|arXiv (Cornell University)|Nov 1, 2021

Advanced Neural Network Applications被引用 9

一句话总结

该论文提出RMNet，一种通过引入保留与合并（RM）操作，等效地从ResNet和MobileNetV2等深层网络中移除残差连接的方法。该操作保留输入特征图，并将其与模块输出合并，从而在不损失精度的前提下实现单分支、类似VGG的架构。RMNet在准确率-速度权衡上达到当前最优表现，其中RMNet 152×6_32在ImageNet上达到80.36%的top-1准确率，超过ResNet-101和RepVGG B2，且高度适用于剪枝与推理优化。

ABSTRACT

Although residual connection enables training very deep neural networks, it is not friendly for online inference due to its multi-branch topology. This encourages many researchers to work on designing DNNs without residual connections at inference. For example, RepVGG re-parameterizes multi-branch topology to a VGG-like (single-branch) model when deploying, showing great performance when the network is relatively shallow. However, RepVGG can not transform ResNet to VGG equivalently because re-parameterizing methods can only be applied to linear blocks and the non-linear layers (ReLU) have to be put outside of the residual connection which results in limited representation ability, especially for deeper networks. In this paper, we aim to remedy this problem and propose to remove the residual connection in a vanilla ResNet equivalently by a reserving and merging (RM) operation on ResBlock. Specifically, the RM operation allows input feature maps to pass through the block while reserving their information and merges all the information at the end of each block, which can remove residual connections without changing the original output. As a plug-in method, RM Operation basically has three advantages: 1) its implementation makes it naturally friendly for high ratio network pruning. 2) it helps break the depth limitation of RepVGG. 3) it leads to better accuracy-speed trade-off network (RMNet) compared to ResNet and RepVGG. We believe the ideology of RM Operation can inspire many insights on model design for the community in the future. Code is available at: https://github.com/fxmeng/RMNet.

研究动机与目标

• 为解决残差连接因多分支拓扑结构导致在线推理效率低下的问题。
• 克服类似RepVGG的重参数化方法在深层网络中无法处理残差模块内非线性层的局限性。
• 实现将预训练的ResNet和MobileNetV2模型转换为无精度损失的纯单分支架构。
• 改善准确率-速度权衡，并增强与网络剪枝及高比例模型压缩的兼容性。
• 探索在深层网络中移除残差连接的潜力，同时保持或提升性能。

提出的方法

• 提出RM操作，通过残差块中的首个卷积、批归一化和ReLU操作保留输入特征图。
• 将保留的输入特征图与残差块最终卷积的输出进行融合，从而有效移除跳跃连接。
• 在每个残差块中应用RM操作，将多分支残差结构转换为单分支、类似VGG的架构。
• 使用逐组卷积和逐点卷积，以最小化保留与融合过程中的参数量和FLOP开销。
• 通过用两组3×3卷积、BN和ReLU替换ResNet的初始主干，设计RMNet，并保持相同的头部和模块结构。
• 采用通道宽度乘数和分组通道设置（例如RMNet 50×6_32）来控制模型容量与效率。

实验结果

研究问题

• RQ1 能否在不改变原始模型输出的前提下，从ResNet等深层网络中移除残差连接？
• RQ2 为何类似RepVGG的重参数化方法在深层网络中难以有效扩展？
• RQ3 单分支、纯网络架构能否在准确率-速度权衡上优于ResNet和RepVGG？
• RQ4 移除残差连接是否能提升与高比例网络剪枝的兼容性？
• RQ5 RM操作能否推广至MobileNetV2等其他架构？

主要发现

• RMNet 50×6_32在ImageNet上达到79.57%的top-1准确率，优于ResNet-101（77.21%）和RepVGG B2（78.78%），且推理速度相当。
• RMNet 101×6_16在ImageNet上达到80.07%的top-1准确率，是首个不依赖架构技巧即突破80%准确率的纯网络模型。
• RMNet 152×6_32在ImageNet上达到80.36%的top-1准确率，证明在RM操作支持下，更深的纯网络依然有效。
• RMNet 26×3_32在CIFAR-10上达到95.81%的准确率，参数量仅880万，FLOPs为0.51 GFLOPs，精度与速度均优于RepVGG A2。
• RM操作对逐点卷积的参数和FLOPs增加仅1/T，且在分组卷积中具有参数效率。
• 由于去除了残差连接，RMNet高度适用于剪枝，可在无精度损失的前提下实现高比例压缩。