[Paper Review] EIE: Efficient Inference Engine on Compressed Deep Neural Network

Song Han, Xingyu Liu+5 more|arXiv (Cornell University)|Feb 4, 2016

Advanced Neural Network Applications140 citations

TL;DR

EIE is a specialized accelerator that runs inference directly on compressed, sparse DNNs with weight sharing, achieving large energy and speedups over CPU/GPU baselines by exploiting activation sparsity, static weight sparsity, and 4-bit weight sharing.

ABSTRACT

State-of-the-art deep neural networks (DNNs) have hundreds of millions of connections and are both computationally and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources and power budgets. While custom hardware helps the computation, fetching weights from DRAM is two orders of magnitude more expensive than ALU operations, and dominates the required power. Previously proposed 'Deep Compression' makes it possible to fit large DNNs (AlexNet and VGGNet) fully in on-chip SRAM. This compression is achieved by pruning the redundant connections and having multiple connections share the same weight. We propose an energy efficient inference engine (EIE) that performs inference on this compressed network model and accelerates the resulting sparse matrix-vector multiplication with weight sharing. Going from DRAM to SRAM gives EIE 120x energy saving; Exploiting sparsity saves 10x; Weight sharing gives 8x; Skipping zero activations from ReLU saves another 3x. Evaluated on nine DNN benchmarks, EIE is 189x and 13x faster when compared to CPU and GPU implementations of the same DNN without compression. EIE has a processing power of 102GOPS/s working directly on a compressed network, corresponding to 3TOPS/s on an uncompressed network, and processes FC layers of AlexNet at 1.88x10^4 frames/sec with a power dissipation of only 600mW. It is 24,000x and 3,400x more energy efficient than a CPU and GPU respectively. Compared with DaDianNao, EIE has 2.9x, 19x and 3x better throughput, energy efficiency and area efficiency.

Motivation & Objective

• Motivate the challenge of deploying large DNNs on embedded hardware due to DRAM energy costs.
• Propose a compression-based approach (pruning and weight sharing) to fit large models in on-chip SRAM.
• Design an accelerator (EIE) that operates directly on compressed networks to accelerate sparse matrix-vector multiplication with weight sharing.
• Demonstrate substantial energy and performance gains over CPUs/GPUs on multiple benchmarks.

Proposed method

• Represent compressed DNN weight matrices using an interleaved compressed sparse column (CSC) format with 4-bit weights and 4-bit indices.
• Distribute the network across multiple processing elements (PEs) by interleaving rows of W to exploit activation sparsity and static weight sparsity.
• Broadcast non-zero input activations and perform scaled multiply-accumulate with 4-bit weights expanded to 16-bit during computation.
• Implement an activation queue and Leading Non-Zero Detection to balance load and enable dynamic sparsity exploitation.
• Provide a central control unit (CCU) and a hierarchical non-zero detection network to coordinate computation and data flow.

Experimental results

Research questions

• RQ1 How much energy and performance benefit can be gained by executing compressed DNNs directly on specialized hardware?
• RQ2 Can the accelerator exploit both static weight sparsity and dynamic activation sparsity along with weight sharing?
• RQ3 What are the design trade-offs (data representation, memory layout, and parallelization) that maximize throughput and energy efficiency for sparse, compressed DNNs?
• RQ4 How does EIE compare to CPU, GPU, and mobile GPU baselines on a variety of networks (AlexNet, VGG-16, NeuralTalk/LSTM) when operating on compressed models?

Key findings

• EIE achieves 189×, 13×, and 307× speedups over CPU, GPU, and mobile GPU, respectively, on nine benchmarks when running compressed networks without batching.
• EIE delivers 102 GOPS/s on compressed networks, equivalent to 3 TOPS/s on uncompressed networks, with 600 mW power dissipation for FC layers of AlexNet.
• Compared with DaDianNao, EIE shows 2.9× throughput, 19× energy efficiency, and 3× area efficiency.
• Overall energy efficiency improvements average 24,000×, 3,400×, and 2,700× over CPU, GPU, and mobile GPU, respectively, driven by SRAM-on-chip, sparsity, and reduced computation.