[论文解读] Joint Neural Architecture Search and Quantization

Yukang Chen, Gaofeng Meng+5 more|arXiv (Cornell University)|Nov 23, 2018

Advanced Neural Network Applications参考文献 29被引用 21

一句话总结

该论文提出JASQ，一种联合神经架构搜索与量化框架，采用多目标进化算法同时优化神经架构与逐层量化策略。该方法在准确率与模型大小之间实现了最先进性能，优于浮点模型及2–16位量化模型，其中JASQNet在CIFAR-10上仅用0.9 MB模型大小即实现2.97%的错误率。

ABSTRACT

Designing neural architectures is a fundamental step in deep learning applications. As a partner technique, model compression on neural networks has been widely investigated to gear the needs that the deep learning algorithms could be run with the limited computation resources on mobile devices. Currently, both the tasks of architecture design and model compression require expertise tricks and tedious trials. In this paper, we integrate these two tasks into one unified framework, which enables the joint architecture search with quantization (compression) policies for neural networks. This method is named JASQ. Here our goal is to automatically find a compact neural network model with high performance that is suitable for mobile devices. Technically, a multi-objective evolutionary search algorithm is introduced to search the models under the balance between model size and performance accuracy. In experiments, we find that our approach outperforms the methods that search only for architectures or only for quantization policies. 1) Specifically, given existing networks, our approach can provide them with learning-based quantization policies, and outperforms their 2 bits, 4 bits, 8 bits, and 16 bits counterparts. It can yield higher accuracies than the float models, for example, over 1.02% higher accuracy on MobileNet-v1. 2) What is more, under the balance between model size and performance accuracy, two models are obtained with joint search of architectures and quantization policies: a high-accuracy model and a small model, JASQNet and JASQNet-Small that achieves 2.97% error rate with 0.9 MB on CIFAR-10.

研究动机与目标

• 自动化神经架构与量化策略的联合设计，以实现高效移动部署。
• 减少模型压缩与架构搜索中的人工经验与手动调优。
• 在准确率与模型大小之间实现优于独立进行架构搜索或量化搜索的平衡。
• 为现有网络提供可学习的、逐层特定的量化策略。

提出的方法

• 使用多目标进化算法探索架构与量化策略的联合搜索空间。
• 适应度评估同时考虑模型准确率与模型大小，由Pareto最优前沿引导搜索过程。
• 通过调节可调阈值$\mathcal{T}_S$的标量化目标函数，支持灵活的权衡调整。
• 种群进化采用锦标赛选择，包含两个超参数：种群大小$\#P$与样本大小$\#S$。
• 在种群中各个体之间共享参数，以加速训练过程。
• 使用Huffman编码表示量化权重，降低存储开销。

实验结果

研究问题

• RQ1 联合搜索神经架构与量化策略是否优于分别优化各部分？
• RQ2 与统一位宽量化相比，逐层自适应量化策略是否能提升准确率？
• RQ3 所提方法能否生成模型紧凑且准确率与全精度模型相当的模型？
• RQ4 联合搜索框架在不同硬件约束下如何平衡准确率与模型大小？

主要发现

• JASQNet在CIFAR-10上仅用0.9 MB模型大小即实现2.97%错误率，准确率优于SqueezeNet，且模型显著更小。
• JASQNet-Small在ImageNet上实现65.90%的top-1准确率，超过SqueezeNet的58.09%，且模型大小相近。
• 量化后的JASQNet模型在MobileNet-v1上准确率比其浮点版本高出超过1.02%。
• 对于ResNet与DenseNet等现有网络，JASQ学习到的量化策略优于2、4、8及16位量化版本。
• 该方法仅需1块GPU运行3天即可完成联合搜索，且现有模型的量化策略可在数小时内找到。
• 搜索过程高效，超参数消融实验表明在$\#P = 16$，$\#S = 16$时性能最优。