[Paper Review] Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks

Philipp Gysel|arXiv (Cornell University)|May 20, 2016

Advanced Neural Network ApplicationsComputer Science38 references101 citations

TL;DR

Ristretto is a fast, GPU-accelerated framework that simulates hardware arithmetic to quantify CNN compression by reducing bit-width of weights and activations, enabling adder-only or low-bitwidth implementations with fine-tuning to maintain accuracy.

ABSTRACT

Convolutional neural networks (CNN) have achieved major breakthroughs in recent years. Their performance in computer vision have matched and in some areas even surpassed human capabilities. Deep neural networks can capture complex non-linear features; however this ability comes at the cost of high computational and memory requirements. State-of-art networks require billions of arithmetic operations and millions of parameters. To enable embedded devices such as smartphones, Google glasses and monitoring cameras with the astonishing power of deep learning, dedicated hardware accelerators can be used to decrease both execution time and power consumption. In applications where fast connection to the cloud is not guaranteed or where privacy is important, computation needs to be done locally. Many hardware accelerators for deep neural networks have been proposed recently. A first important step of accelerator design is hardware-oriented approximation of deep networks, which enables energy-efficient inference. We present Ristretto, a fast and automated framework for CNN approximation. Ristretto simulates the hardware arithmetic of a custom hardware accelerator. The framework reduces the bit-width of network parameters and outputs of resource-intense layers, which reduces the chip area for multiplication units significantly. Alternatively, Ristretto can remove the need for multipliers altogether, resulting in an adder-only arithmetic. The tool fine-tunes trimmed networks to achieve high classification accuracy. Since training of deep neural networks can be time-consuming, Ristretto uses highly optimized routines which run on the GPU. This enables fast compression of any given network. Given a maximum tolerance of 1%, Ristretto can successfully condense CaffeNet and SqueezeNet to 8-bit. The code for Ristretto is available.

Motivation & Objective

• Motivate energy-efficient neural network inference on embedded devices by compressing CNNs without adding decompression complexity.
• Introduce the Ristretto framework to simulate hardware arithmetic and explore bit-width reduction in weights and activations.
• Demonstrate that CNNs like CaffeNet and SqueezeNet can be condensed to 8-bit with minimal accuracy loss under a 1% tolerance.

Proposed method

• Simulate hardware arithmetic of a custom accelerator by quantizing layer inputs, weights, and outputs to reduced precision.
• Support both fixed-point and adder-only arithmetic scenarios by adjusting bit-width and using an adder tree for accumulation.
• Apply rounding schemes (round-nearest-even for inference; stochastic rounding for fine-tuning) to manage quantization errors.
• Fine-tune quantized networks in discrete parameter space using shadow full-precision weights and stochastic rounding during updates.
• Leverage GPU-optimized routines to compress networks quickly without changing network structure or introducing decompression.

Experimental results

Research questions

• RQ1 How does reducing numerical precision of CNN parameters and activations affect classification accuracy under a given tolerance?
• RQ2 Can CNNs such as CaffeNet and SqueezeNet be compressed to 8-bit representations without exceeding a 1% accuracy loss?
• RQ3 What rounding strategies best preserve accuracy during inference and during fine-tuning in discrete parameter spaces?
• RQ4 What are the practical implications of hardware-oriented approximation for memory footprint and multiplier usage in CNN accelerators?

Key findings

• Ristretto can condense CaffeNet and SqueezeNet to 8-bit representations within a 1% tolerance.
• Quantization and fine-tuning in discrete parameter space help recover accuracy after aggressive bit-width reduction.
• Round-nearest-even is used for deterministic inference quantization, while stochastic rounding aids discrete-space fine-tuning.
• The framework targets reducing the memory footprint and multiplier area by simulating hardware arithmetic without introducing decompression overhead.
• Adder-only arithmetic is achievable by adjusting bit-width and accumulation precision in the hardware path.