[Paper Review] Improving Neural Network Quantization without Retraining using Outlier Channel Splitting

Ritchie Zhao, Yuwei Hu+3 more|arXiv (Cornell University)|Jan 28, 2019

Advanced Neural Network Applications31 references150 citations

TL;DR

The paper proposes Outlier Channel Splitting (OCS) to quantize floating-point DNNs without retraining, outperforming clipping with minimal hardware overhead on CNNs and RNNs.

ABSTRACT

Quantization can improve the execution latency and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs, while this work examines the less-studied topic of quantizing a floating-point model without (re)training. DNN weights and activations follow a bell-shaped distribution post-training, while practical hardware uses a linear quantization grid. This leads to challenges in dealing with outliers in the distribution. Prior work has addressed this by clipping the outliers or using specialized hardware. In this work, we propose outlier channel splitting (OCS), which duplicates channels containing outliers, then halves the channel values. The network remains functionally identical, but affected outliers are moved toward the center of the distribution. OCS requires no additional training and works on commodity hardware. Experimental evaluation on ImageNet classification and language modeling shows that OCS can outperform state-of-the-art clipping techniques with only minor overhead.

Motivation & Objective

• Motivate post-training quantization without retraining for real-world deployment.
• Address outlier-driven quantization error by reshaping value distributions.
• Introduce Outlier Channel Splitting (OCS) as a practical, hardware-friendly transform.
• Systematically compare OCS to clipping and other quantization techniques across CNN and RNN models.

Proposed method

• Identify outlier channels with largest absolute values in each layer.
• Duplicate chosen channels and either halve the channel outputs or halve the outgoing weights to preserve functional equivalence (Net2WiderNet transform).
• Apply quantization-aware (QA) splitting to preserve quantized values when splitting.
• Choose number of splits per layer using an expansion ratio r (ceil(r*C) per layer).
• Apply OCS to weights offline and insert a small runtime layer to handle activations on commodity hardware.
• Compare OCS against clipping methods (MSE, ACIQ, KL) for weight and activation quantization across ImageNet CNNs and a language model.

Experimental results

Research questions

• RQ1 Can post-training quantization achieve high accuracy without retraining using channel-level outlier manipulation?
• RQ2 How does OCS compare to clipping-based quantization across weights and activations at various bitwidths?
• RQ3 What is the trade-off between model size overhead and quantization accuracy when applying OCS?
• RQ4 Is quantization-aware splitting necessary to preserve accuracy across bitwidths?
• RQ5 Can OCS extend effectively to RNN-based language models besides CNNs?

Key findings

• OCS can outperform state-of-the-art clipping methods for weight quantization with modest network expansion (small r).
• QA splitting preserves quantization outcomes and outperforms naive Net2WiderNet splitting in activation/weight contexts.
• For activation quantization, clipping generally remains effective, while OCS provides limited gains and is less effective without dynamic channel selection.
• OCS combined with clipping can beat either method alone at very low precision.
• Experiments on ImageNet CNNs and WikiText-2 show that OCS yields improved post-training quantization without retraining, with manageable overhead.
• OCS memory overhead closely tracks the expansion ratio r (approximately 1+r growth in weights/activations).