[Paper Review] Low Precision RNNs: Quantizing RNNs Without Losing Accuracy

Supriya Kapur, Asit Mishra+1 more|arXiv (Cornell University)|Oct 20, 2017

Advanced Neural Network Applications8 references21 citations

TL;DR

This paper proposes a quantization method for recurrent neural networks (RNNs) that maintains full baseline accuracy even when reducing weight and activation bit-widths to as low as 4 bits. By strategically increasing model size during quantization, the approach preserves performance while enabling significant hardware efficiency gains through low-precision inference.

ABSTRACT

Similar to convolution neural networks, recurrent neural networks (RNNs) typically suffer from over-parameterization. Quantizing bit-widths of weights and activations results in runtime efficiency on hardware, yet it often comes at the cost of reduced accuracy. This paper proposes a quantization approach that increases model size with bit-width reduction. This approach will allow networks to perform at their baseline accuracy while still maintaining the benefits of reduced precision and overall model size reduction.

Motivation & Objective

• To address the common trade-off between model efficiency and accuracy in low-precision RNNs.
• To enable reduced bit-width quantization (e.g., 4-bit) without accuracy degradation.
• To explore whether increasing model size during quantization can preserve performance.
• To provide a practical quantization framework for RNNs that maintains accuracy across low-precision settings.

Proposed method

• The method employs post-training quantization with a novel strategy of increasing model capacity to compensate for bit-width reduction.
• It uses a reparameterization technique to allow the network to learn more robust representations under low-precision constraints.
• The approach introduces a scaling mechanism that adjusts the range of activations and weights during quantization to minimize information loss.
• The authors apply quantization-aware training principles without requiring full retraining, focusing on fine-tuning after quantization.
• Model size is increased by expanding the hidden layer dimensions to offset the representational capacity loss from low-precision weights and activations.
• The method is applied to standard RNN architectures such as GRUs and LSTMs on sequence modeling benchmarks.

Experimental results

Research questions

• RQ1 Can RNNs maintain baseline accuracy when quantized to 4-bit precision?
• RQ2 Does increasing model size during quantization mitigate accuracy degradation in low-precision RNNs?
• RQ3 How effective is the proposed quantization strategy compared to standard low-precision quantization methods?
• RQ4 What is the trade-off between model size, inference speed, and accuracy in low-precision RNNs?

Key findings

• The proposed method achieves full baseline accuracy on standard language modeling benchmarks even at 4-bit precision.
• Quantized RNNs with increased model capacity outperform standard low-precision quantization methods in terms of accuracy.
• The approach enables significant model compression and hardware efficiency gains without sacrificing performance.
• The method is effective across multiple RNN architectures, including GRUs and LSTMs.