[Paper Review] FAT: Learning Low-Bitwidth Parametric Representation via Frequency-Aware Transformation

Chaofan Tao, Rui Lin+4 more|arXiv (Cornell University)|Feb 15, 2021

Anomaly Detection Techniques and Applications32 references4 citations

TL;DR

This paper proposes Frequency-Aware Transformation (FAT), a novel quantization framework that improves low-bitwidth CNN training by learning a frequency-domain transform of network weights before quantization. By suppressing redundant high-frequency components and preserving informative low-frequency content, FAT enables simple uniform quantizers to achieve state-of-the-art accuracy—70.5% top-1 on ImageNet for 4-bit ResNet-18 and 69.2% for 4-bit MobileNet-V2—while reducing computation by up to 54.9× compared to full-precision models.

ABSTRACT

Learning convolutional neural networks (CNNs) with low bitwidth is challenging because performance may drop significantly after quantization. Prior arts often discretize the network weights by carefully tuning hyper-parameters of quantization (e.g. non-uniform stepsize and layer-wise bitwidths), which are complicated and sub-optimal because the full-precision and low-precision models have a large discrepancy. This work presents a novel quantization pipeline, Frequency-Aware Transformation (FAT), which has several appealing benefits. (1) Rather than designing complicated quantizers like existing works, FAT learns to transform network weights in the frequency domain before quantization, making them more amenable to training in low bitwidth. (2) With FAT, CNNs can be easily trained in low precision using simple standard quantizers without tedious hyper-parameter tuning. Theoretical analysis shows that FAT improves both uniform and non-uniform quantizers. (3) FAT can be easily plugged into many CNN architectures. When training ResNet-18 and MobileNet-V2 in 4 bits, FAT plus a simple rounding operation already achieves 70.5% and 69.2% top-1 accuracy on ImageNet without bells and whistles, outperforming recent state-of-the-art by reducing 54.9X and 45.7X computations against full-precision models. We hope FAT provides a novel perspective for model quantization. Code is available at \url{https://github.com/ChaofanTao/FAT_Quantization}.

Motivation & Objective

• Address the performance drop in low-bitwidth CNN quantization due to the large capacity gap between full-precision and low-precision models.
• Overcome the limitations of complex quantizers that require extensive hyperparameter tuning and suffer from gradient vanishing in discrete weight spaces.
• Reduce the need for specialized hardware or complex quantization schemes by enabling standard quantizers to work effectively through learned representation transformation.
• Improve training stability and performance in low-bitwidth settings by exploiting inter-neuron correlations via holistic frequency-domain analysis of weights.

Proposed method

• Propose a two-stage quantization pipeline: first applying a learnable frequency-domain transform T(·) to full-precision weights, then applying a standard quantizer Q(·).
• Use Discrete Fourier Transform (DFT) to map spatial-domain weights into the frequency domain, enabling holistic analysis of weight correlations.
• Introduce a soft mask over the frequency spectrum to selectively retain important frequency components and suppress noise-like high-frequency components.
• Train the transform end-to-end using backpropagation, where the gradient is explicitly controlled through the differentiable transform, enabling effective learning despite discretization.
• Use a uniform quantizer after transformation, avoiding complex mixed-precision or adaptive quantization strategies.
• Remove the transform at inference time, leaving only a standard quantizer for deployment, ensuring compatibility with general-purpose hardware.

Experimental results

Research questions

• RQ1 Can a learned frequency-domain transformation improve the performance of simple quantizers in low-bitwidth CNN training?
• RQ2 How does transforming weights in the frequency domain reduce quantization error and bridge the capacity gap between full-precision and low-bitwidth models?
• RQ3 To what extent does the frequency-aware transform improve gradient flow during backpropagation in low-precision training?
• RQ4 Can the proposed method achieve state-of-the-art accuracy with minimal architectural or quantization complexity?
• RQ5 Does the frequency-domain transformation lead to better generalization and robustness in low-bitwidth models compared to direct quantization?

Key findings

• FAT achieves 70.5% top-1 accuracy on ImageNet using 4-bit ResNet-18, outperforming state-of-the-art methods without any hyperparameter tuning or complex quantization schemes.
• For 4-bit MobileNet-V2, FAT achieves 69.2% top-1 accuracy, surpassing recent SOTA results with a 45.7× reduction in computation compared to full-precision models.
• The method reduces model size by up to 10× (e.g., 4.7MB for 3-bit ResNet-18) and BOPs by up to 86.7×, demonstrating high compression efficiency.
• The learned frequency mask suppresses most high-frequency components, effectively denoising the weight representation and improving quantization fidelity.
• Inference on a mobile ARM board shows that uniform quantization after FAT achieves only 398ms latency, significantly faster than complex quantizers like LQ-Net (929ms) or APoT (857ms).
• Theoretical analysis confirms that FAT reduces quantization error and enables informative gradients by exploiting inter-neuron relationships in the frequency domain.