[Paper Review] And the Bit Goes Down: Revisiting the Quantization of Neural Networks

Pierre Stock, Armand Joulin+3 more|arXiv (Cornell University)|Jul 12, 2019

Neural Networks and ApplicationsComputer Science46 references68 citations

TL;DR

The paper presents a activation-focused product quantization method (using weighted k-means) to compress neural networks by quantizing weights with respect to in-domain activations, enabling 20×–26× memory reductions (ResNet-50 and Mask R-CNN) while preserving accuracy and enabling CPU inference.

ABSTRACT

In this paper, we address the problem of reducing the memory footprint of convolutional network architectures. We introduce a vector quantization method that aims at preserving the quality of the reconstruction of the network outputs rather than its weights. The principle of our approach is that it minimizes the loss reconstruction error for in-domain inputs. Our method only requires a set of unlabelled data at quantization time and allows for efficient inference on CPU by using byte-aligned codebooks to store the compressed weights. We validate our approach by quantizing a high performing ResNet-50 model to a memory size of 5MB (20x compression factor) while preserving a top-1 accuracy of 76.1% on ImageNet object classification and by compressing a Mask R-CNN with a 26x factor.

Motivation & Objective

• Reduce memory footprint of convolutional networks, enabling efficient deployment on embedded/CPU devices.
• Preserve network outputs (activations) rather than weights to minimize in-domain reconstruction error.
• Leverage unlabelled data and distillation to learn quantization codebooks in a layer-wise, sequential fashion.
• Provide a practical, byte-aligned quantization scheme that supports CPU inference and broad applicability to detection tasks.

Proposed method

• Apply Product Quantization (PQ) to weight columns by splitting each column into m subvectors and learning a codebook over the subvectors.
• Replace the standard weight reconstruction objective with an activation-focused objective that minimizes ||y − ŷ||^2 for in-domain inputs, implemented as a weighted k-means (E-step using x, M-step via least-squares update).
• Quantize convolutional layers by spatially mapping weight tensors (e.g., using subvector size d = K×K) and process activations accordingly to maximize correlation in the subvectors.
• Quantize the network sequentially from the bottom layer up, using distillation from a non-quantized teacher to fine-tune codewords without requiring labels.
• Globally finetune all codebooks and BatchNorm statistics after layer-wise quantization to reduce drift (9 epochs on ImageNet).
• Provide open-source code and models for replication (GitHub link cited in paper).

Experimental results

Research questions

• RQ1 Can activation-focused quantization outperform weight-focused quantization for in-domain accuracy preservation?
• RQ2 How much memory can be saved on standard ConvNet architectures (e.g., ResNet-50) without compromising top-1 accuracy on ImageNet?
• RQ3 Is sequential layer-wise quantization with distillation effective for complex architectures and detection frameworks (e.g., Mask R-CNN)?
• RQ4 Does using unlabelled data for codebook learning and distillation remove the need for labeled data in quantization?
• RQ5 How does the proposed method compare to contemporary compression approaches across different budgets and tasks?

Key findings

• ResNet-50 compressed to 5 MB (20× reduction) with 76.1% top-1 accuracy on ImageNet using semi-supervised training.
• Mask R-CNN compressed by 26× (to ~6 MB) with competitive Box AP and Mask AP compared to the non-compressed model.
• For vanilla ResNet-18, small-block and large-block setups with k=256 show top-1 around 65.81% and 61.10% respectively (on ~1.54–1.03 MB ranges).
• For vanilla ResNet-50, small-block and large-block setups with k=256 show top-1 around 73.79% and 68.21% respectively (on ~5.09–3.19 MB ranges).
• Semi-supervised ResNet-50 achieves 76.12% top-1 accuracy at ~5.20 MB, matching the performance of a stronger non-compressed baseline within the same size budget.
• Ablation studies indicate activation-aware quantization with distillation consistently outperforms alternatives that use standard PQ or label-based fine-tuning under similar budgets.