[论文解读] ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions

Zechun Liu, Zhiqiang Shen+2 more|arXiv (Cornell University)|Mar 7, 2020

Advanced Neural Network Applications参考文献 44被引用 30

一句话总结

ReActNet 引入一个以 MobileNetV1 为基础的强基线二进制网络，通过通用 ReAct-Sign 和 ReAct-PReLU 对激活分布进行整形，并加入分布损失以更贴近实值输出，在 1-bit CNN 上实现了 ImageNet 的最先进准确率，同时计算量显著降低。

ABSTRACT

In this paper, we propose several ideas for enhancing a binary network to close its accuracy gap from real-valued networks without incurring any additional computational cost. We first construct a baseline network by modifying and binarizing a compact real-valued network with parameter-free shortcuts, bypassing all the intermediate convolutional layers including the downsampling layers. This baseline network strikes a good trade-off between accuracy and efficiency, achieving superior performance than most of existing binary networks at approximately half of the computational cost. Through extensive experiments and analysis, we observed that the performance of binary networks is sensitive to activation distribution variations. Based on this important observation, we propose to generalize the traditional Sign and PReLU functions, denoted as RSign and RPReLU for the respective generalized functions, to enable explicit learning of the distribution reshape and shift at near-zero extra cost. Lastly, we adopt a distributional loss to further enforce the binary network to learn similar output distributions as those of a real-valued network. We show that after incorporating all these ideas, the proposed ReActNet outperforms all the state-of-the-arts by a large margin. Specifically, it outperforms Real-to-Binary Net and MeliusNet29 by 4.0% and 3.6% respectively for the top-1 accuracy and also reduces the gap to its real-valued counterpart to within 3.0% top-1 accuracy on ImageNet dataset. Code and models are available at: https://github.com/liuzechun/ReActNet.

研究动机与目标

• 在 ImageNet 上缩小二值网络与实值网络之间的准确率差距，且不增加额外计算。
• 设计一个高性能的基线二值网络，利用紧凑的实值骨干网络与恒等快捷连接。
• 研究激活分布的影响并引入可学习的广义激活函数以重新塑形分布。
• 使用分布损失使二值网络输出与实值网络对齐。

提出的方法

• 通过对 MobileNetV1 二值化并采用无参数的快捷连接，绕过中间卷积，构建基线二值网络。
• 用基于拼接的设计替换下采样块，以维持相等的输入/输出通道并实现二值化快捷连接。
• 引入 ReAct-Sign (RSign) 和 ReAct-PReLU (RPReLU)，在几乎零成本的情况下学习通道级分布整形与平移。
• 引入分布损失，使实值网络与二值网络输出分布之间的 KL 散度最小化。
• 在 ImageNet 上进行评估，并与最先进的二值网络及实值基线进行比较。

实验结果

研究问题

• RQ1 在 ImageNet 上，1-bit CNN 在尽量减少计算开销的前提下，能在多大程度上接近其实值网络的准确性？
• RQ2 在不增加 FLOPs 的前提下，通过学习分布整形激活和分布损失，是否能显著提升二值网络性能？
• RQ3 下采样设计和激活分布对大规模数据集上二值网络准确性的影响是什么？

主要发现

• 所提出的基线在 ImageNet 上实现 61.1% 的 top-1 准确率，耗费 ~0.87e9 BOPs，优于许多以往的二值网络，成本约为一半。
• 加入 RSign 与 RPReLU 分别比基线提高 4.9% 和 3.6%；两者都使用时（ReActNet-A）达到 69.4% top-1，与实值网络相差不到 3%。
• 分布损失在激活整形之外再提升约 1% 的准确率。
• ReActNet-C 在显著降低 OPs 的情况下达到 71.4% top-1 准确率，超越 MeliusNet59 且更高效。
• 在串联下采样和 1-bit 卷积及快捷连接的基线下，计算成本降低且准确率高于先前设计。
• 可视化显示学习到的激活平移系数类似于实值网络的分布，并带来更均衡的激活图。