[Paper Review] Transform Quantization for CNN (Convolutional Neural Network) Compression

Sean I. Young, Zhe Wang+2 more|arXiv (Cornell University)|Sep 2, 2020

Advanced Neural Network Applications97 references74 citations

TL;DR

This paper proposes transform quantization for post-training compression of CNN weights, optimizing both a learned decorrelating transform and bit-depth allocation via a rate-distortion framework. It achieves state-of-the-art performance, enabling efficient low-bit-rate compression (1–2 bits) of models like AlexNet, ResNet, and DenseNet with minimal accuracy loss.

ABSTRACT

In this paper, we compress convolutional neural network (CNN) weights post-training via transform quantization. Previous CNN quantization techniques tend to ignore the joint statistics of weights and activations, producing sub-optimal CNN performance at a given quantization bit-rate, or consider their joint statistics during training only and do not facilitate efficient compression of already trained CNN models. We optimally transform (decorrelate) and quantize the weights post-training using a rate-distortion framework to improve compression at any given quantization bit-rate. Transform quantization unifies quantization and dimensionality reduction (decorrelation) techniques in a single framework to facilitate low bit-rate compression of CNNs and efficient inference in the transform domain. We first introduce a theory of rate and distortion for CNN quantization, and pose optimum quantization as a rate-distortion optimization problem. We then show that this problem can be solved using optimal bit-depth allocation following decorrelation by the optimal End-to-end Learned Transform (ELT) we derive in this paper. Experiments demonstrate that transform quantization advances the state of the art in CNN compression in both retrained and non-retrained quantization scenarios. In particular, we find that transform quantization with retraining is able to compress CNN models such as AlexNet, ResNet and DenseNet to very low bit-rates (1-2 bits).

Motivation & Objective

• To address the suboptimal performance of existing CNN quantization methods that ignore joint statistics of weights and activations.
• To enable efficient, low-bit-rate compression of already-trained CNN models without retraining.
• To unify dimensionality reduction, quantization, and pruning within a single rate-distortion optimized framework.
• To derive an end-to-end learned transform (ELT) that maximizes compression gains by decorrelating weights and enabling optimal bit-depth allocation.

Proposed method

• Formulates CNN weight compression as a rate-distortion optimization problem, minimizing output distortion under a bit-rate constraint.
• Applies a learned decorrelating transform (ELT) to weight matrices to reduce redundancy and enable efficient quantization.
• Optimizes bit-depth allocation across transform coefficients to minimize distortion, using first-order Taylor approximation to link weight perturbations to output error.
• Uses a transform domain representation where insignificant coefficients are quantized to zero, emulating pruning.
• Derives the optimal transform as a generalized eigenvalue decomposition of the product of activation and weight covariance matrices.
• Supports mixed-precision inference by assigning different bit-depths to different transform channels.

Experimental results

Research questions

• RQ1 Can joint statistics of weights and activations be leveraged post-training to improve CNN compression efficiency?
• RQ2 Does optimizing both transform and bit-depth allocation yield better compression than scalar quantization alone?
• RQ3 Can a learned transform outperform classical transforms like DCT or KLT in low-bit-rate CNN compression?
• RQ4 How does the proposed framework perform in both retrained and non-retrained settings across diverse CNN architectures?
• RQ5 What is the theoretical relationship between the optimal transform and classical transforms like KLT or SVD?

Key findings

• Transform quantization achieves state-of-the-art compression performance for both retrained and non-retrained CNNs, including AlexNet, ResNet, and DenseNet.
• With retraining, the method compresses models to 1–2 bits per weight while maintaining high accuracy, significantly outperforming prior methods.
• The end-to-end learned transform (ELT) achieves gains comparable to the theoretical optimum, with gains of up to 19.8 dB in intra-kernel transform coding for AlexNet.
• The optimal bit-depth allocation, derived from minimizing output distortion, outperforms uniform bit allocation and leads to better performance at low bit-rates.
• The framework enables efficient inference in the transform domain and supports mixed-precision deployment on specialized hardware.
• Theoretical analysis shows that the optimal transform is equivalent to a generalized eigenvalue decomposition of the product of weight and activation covariance matrices.