[论文解读] Deep Adaptive Network: An Efficient Deep Neural Network with Sparse Binary Connections

Xichuan Zhou, Shengli Li+6 more|arXiv (Cornell University)|Apr 21, 2016

Advanced Neural Network Applications参考文献 27被引用 9

一句话总结

本文提出深度自适应网络（Deep Adaptive Network），一种硬件高效的深度神经网络，通过自适应剪枝微小权重并二值化显著权重至±1，实现稀疏二值连接。该方法在保持MNIST数据集97.2%准确率的同时，将内存使用量减少99.3%，计算量减少99.9%，与标准深度置信网络相比仅造成极小的准确率损失。

ABSTRACT

Deep neural networks are state-of-the-art models for understanding the content of images, video and raw input data. However, implementing a deep neural network in embedded systems is a challenging task, because a typical deep neural network, such as a Deep Belief Network using 128x128 images as input, could exhaust Giga bytes of memory and result in bandwidth and computing bottleneck. To address this challenge, this paper presents a hardware-oriented deep learning algorithm, named as the Deep Adaptive Network, which attempts to exploit the sparsity in the neural connections. The proposed method adaptively reduces the weights associated with negligible features to zero, leading to sparse feedforward network architecture. Furthermore, since the small proportion of important weights are significantly larger than zero, they can be robustly thresholded and represented using single-bit integers (-1 and +1), leading to implementations of deep neural networks with sparse and binary connections. Our experiments showed that, for the application of recognizing MNIST handwritten digits, the features extracted by a two-layer Deep Adaptive Network with about 25% reserved important connections achieved 97.2% classification accuracy, which was almost the same with the standard Deep Belief Network (97.3%). Furthermore, for efficient hardware implementations, the sparse-and-binary-weighted deep neural network could save about 99.3% memory and 99.9% computation units without significant loss of classification accuracy for pattern recognition applications.

研究动机与目标

• 解决深度神经网络在嵌入式系统中高内存与计算需求的问题。
• 在不造成显著准确率下降的前提下，减小模型尺寸与推理成本。
• 通过稀疏性与二值权重表示，实现高效的硬件部署。
• 开发一种可训练的自适应剪枝机制，以保留重要特征。

提出的方法

• 自适应地将与微小特征相关的权重设为零，从而在网络中引入结构稀疏性。
• 采用阈值化机制仅保留最显著的权重，随后将其二值化为±1。
• 使用可微剪枝策略，支持带有稀疏性正则化的端到端训练。
• 采用两阶段训练流程：首先使用标准反向传播进行预训练，然后在稀疏性与二值化约束下进行微调。
• 实现一种适合低功耗硬件的稀疏二值权重前馈架构。
• 引入重参数化技巧，以在二值权重训练过程中稳定梯度。

实验结果

研究问题

• RQ1 自适应剪枝与二值量化是否能保留深度神经网络的分类准确率？
• RQ2 稀疏性与二值权重在多大程度上可减少深度网络的内存与计算量？
• RQ3 在基准数据集上，该方法与标准深度置信网络相比准确率如何？
• RQ4 在稀疏二值网络中，模型压缩与性能损失之间的权衡如何？
• RQ5 该方法是否能在保持高准确率的同时实现有效的端到端训练？

主要发现

• 深度自适应网络在MNIST数据集上仅保留25%的连接，即实现了97.2%的分类准确率。
• 与全精度网络相比，该模型将内存使用量减少了约99.3%。
• 由于稀疏且二值化的权重表示，计算操作减少了99.9%。
• 性能与标准深度置信网络几乎完全一致，后者达到97.3%的准确率。
• 尽管权重被极端量化至±1，该方法仍保持了高度的鲁棒性与泛化能力。
• 该方法在资源受限的嵌入式系统中部署方面展现出强大潜力。