[Paper Review] WRPN: Wide Reduced-Precision Networks

Asit Mishra, Eriko Nurvitadhi+2 more|arXiv (Cornell University)|Sep 4, 2017

Advanced Neural Network Applications20 references181 citations

TL;DR

WRPN trains and runs deep nets with wide layers and reduced-precision activations and weights, achieving at-par or better accuracy than full-precision baselines while significantly reducing memory, bandwidth, and energy, with hardware-friendly quantization and results on AlexNet, ResNet-34, and Inception-BN variants.

ABSTRACT

For computer vision applications, prior works have shown the efficacy of reducing numeric precision of model parameters (network weights) in deep neural networks. Activation maps, however, occupy a large memory footprint during both the training and inference step when using mini-batches of inputs. One way to reduce this large memory footprint is to reduce the precision of activations. However, past works have shown that reducing the precision of activations hurts model accuracy. We study schemes to train networks from scratch using reduced-precision activations without hurting accuracy. We reduce the precision of activation maps (along with model parameters) and increase the number of filter maps in a layer, and find that this scheme matches or surpasses the accuracy of the baseline full-precision network. As a result, one can significantly improve the execution efficiency (e.g. reduce dynamic memory footprint, memory bandwidth and computational energy) and speed up the training and inference process with appropriate hardware support. We call our scheme WRPN - wide reduced-precision networks. We report results and show that WRPN scheme is better than previously reported accuracies on ILSVRC-12 dataset while being computationally less expensive compared to previously reported reduced-precision networks.

Motivation & Objective

• Motivate memory and compute efficiency challenges in training/inferring DNNs for vision tasks, focusing on activation maps as a major memory footprint.
• Propose WRPN: reduce both activations and weights while widening layer width to preserve accuracy.
• Demonstrate that wide, low-precision networks can match or surpass baseline accuracy across several architectures on ImageNet.
• Assess hardware implications and potential efficiency gains on GPUs, FPGAs, and ASICs.

Proposed method

• Quantize activations to 4-bit or 2-bit and weights to 2-bit or 1- or 4-bit per layer using a simple clipping-and-rounding scheme.
• Increase the number of filter maps per layer (width) to compensate for information loss due to low precision, preserving or improving accuracy.
• Train networks end-to-end from scratch with WRPN quantization and width, comparing to full-precision baselines across AlexNet, ResNet-34, and Inception-BN variants.
• Use a straight-through estimator (STE) for backpropagation through quantized nodes, with simple min-max clipping and rounding for k-bit representations (k>1) and BWN-like handling for binary cases.
• Evaluate compute cost as the product of the number of FMA operations and the sum of activation and weight widths, and compare to baseline FP32 compute cost.

Experimental results

Research questions

• RQ1 Can activation map precision be reduced below full precision without harming accuracy when the network width is increased?
• RQ2 Does widening the network while using low-precision operands compensate for information loss and maintain or improve accuracy?
• RQ3 What are the accuracy and hardware efficiency trade-offs for WRPN across standard vision architectures on ImageNet?
• RQ4 How do WRPN configurations perform on deeper networks like ResNet-34 and Inception-BN compared to AlexNet?
• RQ5 What is the practical hardware performance (GPU/FPGA/ASIC) implications of WRPN’s low-precision regimes?

Key findings

• Activations at 4-bit and weights at 2-bit can match baseline accuracy on AlexNet when the network is widened.
• Doubling the number of filters with 4-bit activations and 2-bit weights yields AlexNet accuracy at par with the full-precision baseline (Tables show 4b A/2b W with 2x-wide achieves at-par accuracy).
• For ResNet-34, 2-bit weights and 4-bit activations continue to provide at-par accuracy with 2x-wide expansion; 4-bit A and 2-bit W also achieves at-par, while 2-bit A and W (binary/ternary) variants can approach baseline with larger width.
• In batch-normalized Inception, 4-bit activations and 4-bit weights with doubling of filter banks achieves near-baseline accuracy (71.63 vs 71.64).
• Widening with WRPN typically increases raw compute operations, but lower operand bit-width delivers substantial efficiency gains; FPGA and ASIC offer favorable efficiency (6.5x to 100x) over FP32, while GPU gains are more modest.
• Binary/ternary configurations with wider networks can achieve near-baseline or state-of-the-art accuracy for ResNet-34 and AlexNet while reducing compute cost significantly.
• Across networks, 4-bit activations and 2-bit weights emerge as a practical strong operating point, balancing accuracy and hardware simplicity (e.g., enabling multipliers-less implementations with ternary weights).
• WRPN demonstrates clear hardware-friendliness: FPGA and ASIC show large efficiency gains; GPU shows more limited gains due to limited native low-precision support.