[Paper Review] An Area and Energy Efficient Design of Domain-Wall Memory-Based Deep Convolutional Neural Networks using Stochastic Computing

Xiaolong Ma, Yipeng Zhang+6 more|arXiv (Cornell University)|Feb 3, 2018

Error Correcting Code Techniques28 references3 citations

TL;DR

This paper proposes DW-CNN, the first comprehensive optimization framework for area- and energy-efficient deep convolutional neural networks (DCNNs) using stochastic computing (SC) and domain-wall memory (DWM). By replacing SRAM with high-density, non-volatile DWM for weight storage and optimizing memory precision, organization, and resource sharing, the framework reduces area by up to 52.6% and power consumption by 17.35× while maintaining high application-level accuracy (e.g., 99.01% on LeNet-5 with 1024-bit stochastic streams).

ABSTRACT

With recent trend of wearable devices and Internet of Things (IoTs), it becomes attractive to develop hardware-based deep convolutional neural networks (DCNNs) for embedded applications, which require low power/energy consumptions and small hardware footprints. Recent works demonstrated that the Stochastic Computing (SC) technique can radically simplify the hardware implementation of arithmetic units and has the potential to satisfy the stringent power requirements in embedded devices. However, in these works, the memory design optimization is neglected for weight storage, which will inevitably result in large hardware cost. Moreover, if conventional volatile SRAM or DRAM cells are utilized for weight storage, the weights need to be re-initialized whenever the DCNN platform is re-started. In order to overcome these limitations, in this work we adopt an emerging non-volatile Domain-Wall Memory (DWM), which can achieve ultra-high density, to replace SRAM for weight storage in SC-based DCNNs. We propose DW-CNN, the first comprehensive design optimization framework of DWM-based weight storage method. We derive the optimal memory type, precision, and organization, as well as whether to store binary or stochastic numbers. We present effective resource sharing scheme for DWM-based weight storage in the convolutional and fully-connected layers of SC-based DCNNs to achieve a desirable balance among area, power (energy) consumption, and application-level accuracy.

Motivation & Objective

• Address the high hardware cost and energy consumption of memory in SC-based DCNNs, especially for weight storage in fully-connected layers.
• Overcome the limitations of volatile SRAM/DRAM, which require re-initialization after power-up, by adopting non-volatile domain-wall memory (DWM).
• Design a comprehensive optimization framework for DWM-based weight storage in SC-based DCNNs to minimize area and energy while preserving application-level accuracy.
• Investigate optimal memory type, precision, organization, and data format (binary vs. stochastic) for weight representation in DWM.
• Develop an effective resource sharing scheme for DWM-based weight storage in convolutional and fully-connected layers to balance area, power, and accuracy.

Proposed method

• Adopt domain-wall memory (DWM), a high-density, non-volatile spintronic memory, to replace conventional SRAM for weight storage in SC-based DCNNs.
• Optimize the memory organization by evaluating different precision levels (e.g., 7-bit binary, 32–1024-bit stochastic streams) and memory types to minimize area and energy.
• Implement a resource sharing scheme for DWM-based weight storage in both convolutional and fully-connected layers to reduce redundant circuitry.
• Use stochastic computing (SC) to simplify arithmetic units, reducing area and power consumption in computation blocks like inner product and activation functions.
• Integrate DWM read/write circuitry sharing across multiple weight matrices to minimize area and energy overhead in fully-connected layers.
• Evaluate the framework on LeNet-5 using the MNIST dataset, with hardware parameters derived from CACTI (SRAM) and Synopsys Design Compiler (logic), and DWM parameters from prior literature.

Experimental results

Research questions

• RQ1 What is the optimal memory type (DWM vs. SRAM) for storing weights in SC-based DCNNs in terms of area and energy efficiency?
• RQ2 How does the choice between storing weights as binary or stochastic numbers affect area, power, and application-level accuracy in DWM-based SC-DCNNs?
• RQ3 What is the optimal precision (bit-stream length) for stochastic weight representation that maintains high accuracy while minimizing hardware cost?
• RQ4 How effective is resource sharing in reducing area and power consumption for DWM-based weight storage in fully-connected layers?
• RQ5 To what extent can DWM-based weight storage enable a non-volatile, plug-and-play DCNN platform without re-initialization after power-up?

Key findings

• Using DWM for weight storage in SC-based DCNNs reduces area by up to 52.6% and power consumption by 17.35× compared to non-shared DWM configurations in the fully-connected layer.
• Storing weights as stochastic numbers in DWM reduces area and power consumption more effectively than binary representation, especially as bit-stream length increases.
• Application-level accuracy remains high (99.01% on LeNet-5) even with 1024-bit stochastic weight streams, and 256-bit streams yield 98.95% accuracy, supporting the use of shorter streams for efficiency.
• Power consumption increases significantly with SRAM and longer bit-streams, but remains nearly flat with DWM due to its low leakage and high density.
• The resource sharing scheme reduces area and power by minimizing the number of DWM read/write circuits and APC (accumulator-based product) blocks.
• DWM enables non-volatile weight storage, eliminating the need for re-initialization after power-up and enabling true plug-and-play operation for embedded DCNN systems.