[Paper Review] Hardware-oriented Approximation of Convolutional Neural Networks

Philipp Gysel, Mohammad Motamedi+1 more|arXiv (Cornell University)|Apr 11, 2016

Advanced Neural Network Applications21 references280 citations

TL;DR

Ristretto is a Caffe-based framework that automatically trims floating-point CNNs to fixed-point with dynamic fixed point representations, then fine-tunes them, achieving 8-bit models with minimal accuracy loss under a 1% error tolerance.

ABSTRACT

High computational complexity hinders the widespread usage of Convolutional Neural Networks (CNNs), especially in mobile devices. Hardware accelerators are arguably the most promising approach for reducing both execution time and power consumption. One of the most important steps in accelerator development is hardware-oriented model approximation. In this paper we present Ristretto, a model approximation framework that analyzes a given CNN with respect to numerical resolution used in representing weights and outputs of convolutional and fully connected layers. Ristretto can condense models by using fixed point arithmetic and representation instead of floating point. Moreover, Ristretto fine-tunes the resulting fixed point network. Given a maximum error tolerance of 1%, Ristretto can successfully condense CaffeNet and SqueezeNet to 8-bit. The code for Ristretto is available.

Motivation & Objective

• Motivate hardware-friendly CNN deployment by reducing computation and memory demands.
• Develop a post-training quantization and fine-tuning tool to convert floating-point CNNs to fixed-point representations.
• Explore dynamic fixed point as a strategy to preserve accuracy across large networks.
• Quantify how bit-width reduction impacts accuracy across convolutional and fully connected layers.
• Provide an open-source solution usable with Caffe for accelerator-oriented CNN compression.

Proposed method

• Quantize floating-point networks to fixed point with round-nearest mapping.
• Analyze layer activations to determine sufficient integer part to avoid saturation.
• Perform a binary search to find optimal bit-widths for convolution weights, layer outputs, and fully connected weights.
• Split each network layer into two groups (weights and outputs) with independent bit-widths to manage dynamic range.
• Retrain the fixed-point network (fine-tuning) using full-precision shadow weights and stochastic rounding.
• Use floating-point accumulation during forward propagation for hardware-simulated precision to leverage optimized matrix operations.

Experimental results

Research questions

• RQ1 Can fixed-point representations (with dynamic fixed point) maintain high accuracy when compressing large CNNs?
• RQ2 What are the trade-offs between bit-width reduction and classification accuracy for convolutional versus fully connected layers?
• RQ3 How does dynamic fixed point compare to static fixed point in preserving accuracy for large networks like CaffeNet and GoogLeNet?
• RQ4 What is the impact of post-training quantization and subsequent fine-tuning on network performance across several architectures?
• RQ5 Is it feasible to achieve 8-bit or lower representations for modern CNNs without significant accuracy loss?

Key findings

• Dynamic fixed point maintains accuracy significantly better than static fixed point at lower bit-widths for large networks.
• Convolutional kernels and layer activations can be quantized to 8-bit with about 0.3% absolute accuracy loss across networks.
• Fully connected layer parameters are more sensitive, with up to 0.9% absolute accuracy loss at 8-bit.
• LeNet weights can be trimmed to 2-bit with under 0.4% accuracy loss.
• Five networks (LeNet, CIFAR-10, CaffeNet, GoogLeNet, SqueezeNet) can be fine-tuned to 8-bit dynamic fixed point with less than 2.3% accuracy drop from the floating-point baselines.
• Ristretto enables substantial hardware benefits, including ~4–8x memory bandwidth reduction and 4–8x more parameters on-chip, without decompression overhead.