[Paper Review] Overwrite Quantization: Opportunistic Outlier Handling for Neural Network Accelerators.

Ritchie Zhao, Christopher De+1 more|arXiv (Cornell University)|Oct 13, 2019

Model Reduction and Neural Networks11 references4 citations

TL;DR

Overwrite Quantization (OverQ) is a hardware technique that opportunistically increases bitwidth for outlier weights and activations by overwriting adjacent values, enabling accurate 4-bit inference in DNN accelerators with minimal area overhead. An FPGA prototype demonstrates significant accuracy improvements for ResNet-18 with only marginal resource cost.

ABSTRACT

Outliers in weights and activations pose a key challenge for fixed-point quantization of neural networks. While outliers can be addressed by fine-tuning, this is not practical for machine learning (ML) service providers (e.g., Google, Microsoft) who often receive customers' models without the training data. Specialized hardware for handling outliers can enable low-precision DNNs, but incurs nontrivial area overhead. In this paper, we propose overwrite quantization (OverQ), a novel hardware technique which opportunistically increases bitwidth for outliers by letting them overwrite adjacent values. An FPGA prototype shows OverQ can significantly improve ResNet-18 accuracy at 4 bits while incurring relatively little increase in resource utilization.

Motivation & Objective

• Address the challenge of outliers in fixed-point quantization of neural networks, which degrade model accuracy when using low-precision formats.
• Enable practical low-precision inference in machine learning service providers' environments where training data is unavailable for fine-tuning.
• Minimize hardware area overhead while supporting outlier handling in neural network accelerators.
• Design a scalable and efficient hardware solution that integrates seamlessly into existing DNN accelerator pipelines.

Proposed method

• Overwrite Quantization dynamically increases bitwidth for outlier values by overwriting neighboring values in the weight or activation memory layout.
• The technique identifies outliers during quantization and reassigns their storage to adjacent locations with extended bitwidth, preserving data integrity.
• A hardware-aware quantization scheme integrates outlier detection and bitwidth adjustment during the inference phase without requiring retraining.
• The approach uses a compact control logic to manage bitwidth switching and memory mapping, minimizing area overhead.
• An FPGA prototype implements OverQ in a DNN accelerator, validating its effectiveness and efficiency on ResNet-18.

Experimental results

Research questions

• RQ1 Can outlier handling in low-precision DNN inference be achieved without retraining or access to training data?
• RQ2 How can hardware support for outliers be implemented with minimal area overhead in neural network accelerators?
• RQ3 To what extent does OverQ improve model accuracy at 4-bit precision compared to standard quantization?
• RQ4 Can the overwrite mechanism maintain high throughput and low latency in real-time inference workloads?

Key findings

• OverQ significantly improves ResNet-18 accuracy at 4-bit quantization, outperforming standard quantization methods.
• The FPGA prototype shows that OverQ incurs only a marginal increase in resource utilization despite handling outliers.
• The technique enables accurate inference without requiring fine-tuning, making it suitable for ML service providers with restricted access to training data.
• By overwriting adjacent values, OverQ achieves effective outlier bitwidth expansion with minimal hardware complexity.