[论文解读] A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification

Babak Rokh, Ali Azarpeyvand+1 more|arXiv (Cornell University)|May 14, 2022

Advanced Neural Network Applications参考文献 175被引用 5

一句话总结

本综述对图像分类中深度神经网络的模型量化技术进行了全面分析，涵盖基于聚类的方法、缩放因子近似、使用直通估计器的训练策略以及低精度计算。在CIFAR-10和ImageNet上评估了最先进方法，统一比较了不同量化方法在精度、效率和硬件权衡方面的表现。

ABSTRACT

Recent advancements in machine learning achieved by Deep Neural Networks (DNNs) have been significant. While demonstrating high accuracy, DNNs are associated with a huge number of parameters and computations, which leads to high memory usage and energy consumption. As a result, deploying DNNs on devices with constrained hardware resources poses significant challenges. To overcome this, various compression techniques have been widely employed to optimize DNN accelerators. A promising approach is quantization, in which the full-precision values are stored in low bit-width precision. Quantization not only reduces memory requirements but also replaces high-cost operations with low-cost ones. DNN quantization offers flexibility and efficiency in hardware design, making it a widely adopted technique in various methods. Since quantization has been extensively utilized in previous works, there is a need for an integrated report that provides an understanding, analysis, and comparison of different quantization approaches. Consequently, we present a comprehensive survey of quantization concepts and methods, with a focus on image classification. We describe clustering-based quantization methods and explore the use of a scale factor parameter for approximating full-precision values. Moreover, we thoroughly review the training of a quantized DNN, including the use of a straight-through estimator and quantization regularization. We explain the replacement of floating-point operations with low-cost bitwise operations in a quantized DNN and the sensitivity of different layers in quantization. Furthermore, we highlight the evaluation metrics for quantization methods and important benchmarks in the image classification task. We also present the accuracy of the state-of-the-art methods on CIFAR-10 and ImageNet.

研究动机与目标

• 提供深度神经网络在图像分类中模型量化技术的统一、系统性综述。
• 分析并比较基于聚类的量化与缩放因子近似方法在低精度参数表示中的表现。
• 研究使用直通估计器和量化正则化等训练策略，以在低比特网络中保持精度。
• 评估量化对层敏感性的影响，以及将浮点数运算替换为按位运算的影响。
• 在CIFAR-10和ImageNet等标准数据集上对最先进量化方法进行基准测试，报告精度和效率指标。

提出的方法

• 将量化方法分类为基于聚类的方法和基于缩放因子的近似方法，用于将全精度权重映射到低比特表示。
• 回顾直通估计器（STE）在训练过程中通过离散量化操作反向传播梯度的应用。
• 分析量化正则化技术，以稳定低精度网络的训练并提升泛化能力。
• 研究在推理流程中用低成本按位运算替代高成本浮点运算的可行性。
• 研究各层对量化的敏感性，识别出哪些层对精度更鲁棒或更关键。
• 使用标准基准评估量化性能，包括ImageNet和CIFAR-10上的top-1和top-5精度。

实验结果

研究问题

• RQ1 基于聚类和基于缩放因子的量化方法在精度和参数效率方面如何比较？
• RQ2 直通估计器在实现量化网络端到端训练中起到什么作用？
• RQ3 量化正则化如何提升低精度模型的泛化能力和鲁棒性？
• RQ4 DNN中的哪些层对量化最敏感，这对模型精度有何影响？
• RQ5 在CIFAR-10和ImageNet基准测试中，最先进量化模型的精度结果如何？

主要发现

• 综述指出，在大多数情况下，采用逐层或逐通道校准的基于缩放因子的量化方法，其精度高于均匀量化。
• 使用直通估计器可实现有效反向传播通过量化操作，从而支持低精度网络的端到端训练。
• 量化正则化显著提升收敛速度和最终精度，尤其在低比特范围（如4比特或更低）表现更明显。
• 大卷积核尺寸的卷积层和全连接层对量化更敏感，需要仔细校准或微调。
• 最先进量化模型在CIFAR-10上实现超过90%的top-1精度，在ImageNet上使用8比特量化时达到超过75%的top-1精度，接近全精度性能。
• 集成低比特宽操作（如4比特或8比特）可将内存使用量减少最多4倍，并在边缘设备上将推理速度提升2至3倍。