[Paper Review] ApproxDBN: Approximate Computing for Discriminative Deep Belief Networks

Xiaojing Xu, Srinjoy Das+1 more|arXiv (Cornell University)|Apr 13, 2017

Low-power high-performance VLSI design11 references3 citations

TL;DR

This paper proposes ApproxDBN, a framework for energy-efficient inference in Discriminative Deep Belief Networks (DDBNs) using approximate computing. By combining criticality analysis to identify resilient neurons, iterative bit-length reduction, and incremental retraining, the method achieves up to 90% power savings with less than 10% accuracy loss on MNIST.

ABSTRACT

Probabilistic generative neural networks are useful for many applications, such as image classification, speech recognition and occlusion removal. However, the power budget for hardware implementations of neural networks can be extremely tight. To address this challenge we describe a design methodology for using approximate computing methods to implement Approximate Deep Belief Networks (ApproxDBNs) by systematically exploring the use of (1) limited precision of variables; (2) criticality analysis to identify the nodes in the network which can operate with such limited precision while allowing the network to maintain target accuracy levels; and (3) a greedy search methodology with incremental retraining to determine the optimal reduction in precision to enable maximize power savings under user-specified accuracy constraints. Experimental results show that significant bit-length reduction can be achieved by our ApproxDBN with constrained accuracy loss.

Motivation & Objective

• Address the challenge of high power consumption in hardware implementations of deep belief networks for mobile and IoT devices.
• Leverage the inherent error resilience of neural networks to reduce variable precision without significant accuracy degradation.
• Develop a systematic methodology for approximating DDBNs that balances power savings and accuracy constraints.
• Optimize bit-length distribution across network variables to minimize power consumption while maintaining user-specified accuracy levels.

Proposed method

• Perform criticality analysis on a full-precision DDBN to rank hidden neurons by their impact on network performance.
• Apply iterative, greedy bit-length reduction to the least critical neurons based on a dynamically adjusted threshold.
• Use incremental retraining after each approximation step to recover and improve network accuracy.
• Repeat the approximation-retraining cycle until further reduction violates the accuracy constraint.
• Employ a three-level iterative framework: (1) criticality ordering, (2) neuron selection with adaptive thresholding, and (3) progressive bit-width reduction.
• Utilize Q8.8 fixed-point representation as baseline and reduce variable precision to lower bit-lengths (e.g., 8-bit, 4-bit, 2-bit) for selected neurons.

Experimental results

Research questions

• RQ1 How can variable precision reduction be systematically applied to generative, stochastic deep belief networks to minimize power consumption?
• RQ2 Which neurons in a DDBN are most resilient to precision reduction, and how can they be identified before approximation?
• RQ3 To what extent can retraining after approximation restore or improve network accuracy after bit-width reduction?
• RQ4 What is the optimal bit-length distribution across DDBN parameters that satisfies a given accuracy constraint with minimal power cost?
• RQ5 How does the combination of criticality analysis and iterative retraining outperform random or fixed-precision approximation strategies?

Key findings

• Criticality-based neuron selection outperforms random selection, maintaining higher accuracy when approximating the same number of neurons in multi-layer DDBNs.
• For a 300-200-100 DDBN, the ApproxDBN framework achieved a 90% accuracy level with a 90% reduction in total bit-length compared to baseline.
• Retraining after each approximation step significantly improved accuracy recovery, especially when larger precision reductions were applied.
• The greedy, iterative design flow reduced total bit-length monotonically across six iterations while maintaining accuracy above the 90% threshold.
• The framework enabled up to 90% power savings with less than 10% accuracy loss on the MNIST dataset, demonstrating strong energy-efficiency trade-offs.
• The method achieved higher bit-length reduction and better accuracy retention than variants without criticality analysis or retraining.