[Paper Review] Learning Machines Implemented on Non-Deterministic Hardware

Suyog Gupta, Vikas Sindhwani+1 more|arXiv (Cornell University)|Sep 9, 2014

Low-power high-performance VLSI design24 references3 citations

TL;DR

This paper proposes accelerating machine learning workloads by offloading key matrix computations to non-deterministic, stochastic hardware that trades precision for speed and energy efficiency. Using digital stochastic circuits to approximate matrix multiplication, the authors demonstrate that deep neural networks trained on this noisy hardware achieve comparable or slightly improved test accuracy compared to precise computation, validating the feasibility of error-tolerant hardware-software co-design for large-scale learning systems.

ABSTRACT

This paper highlights new opportunities for designing large-scale machine learning systems as a consequence of blurring traditional boundaries that have allowed algorithm designers and application-level practitioners to stay -- for the most part -- oblivious to the details of the underlying hardware-level implementations. The hardware/software co-design methodology advocated here hinges on the deployment of compute-intensive machine learning kernels onto compute platforms that trade-off determinism in the computation for improvement in speed and/or energy efficiency. To achieve this, we revisit digital stochastic circuits for approximating matrix computations that are ubiquitous in machine learning algorithms. Theoretical and empirical evaluation is undertaken to assess the impact of the hardware-induced computational noise on algorithm performance. As a proof-of-concept, a stochastic hardware simulator is employed for training deep neural networks for image recognition problems.

Motivation & Objective

• To explore the feasibility of using non-deterministic, approximate hardware for accelerating large-scale machine learning workloads.
• To address the growing performance and energy efficiency gap in traditional general-purpose processors as Moore’s Law slows.
• To design a hardware-software co-design framework that leverages the inherent noise resilience of machine learning algorithms.
• To evaluate whether stochastic hardware-induced noise can be harnessed to improve or maintain learning performance without sacrificing accuracy.
• To enable practical deployment of such accelerators by preserving the programming model and minimizing software development overhead.

Proposed method

• The authors use digital stochastic circuits to represent numbers as random bit sequences, enabling approximate computation of matrix multiplications central to machine learning.
• They introduce a high-level abstraction to model the statistical error introduced by stochastic computation, particularly in matrix multiplication kernels.
• The framework integrates stochastic hardware accelerators into the training pipeline, where they execute compute-intensive operations like backpropagation.
• The system employs batch gradient descent with a tunable stochastic bit-sequence length N as a new hyperparameter to control noise levels.
• The host processor manages the accelerator and coordinates training, while the stochastic hardware introduces controlled noise during forward and backward passes.
• The approach is evaluated using deep neural networks on the MNIST dataset, with training conducted under varying noise levels defined by N.

Experimental results

Research questions

• RQ1 Can non-deterministic, approximate hardware be used to accelerate machine learning without degrading model accuracy?
• RQ2 How does hardware-induced computational noise from stochastic circuits affect the convergence and performance of gradient-based learning algorithms?
• RQ3 Does the noise introduced by stochastic computation act as a regularizer, improving generalization in deep neural networks?
• RQ4 What is the impact of stochastic bit-sequence length N on training stability and final model accuracy?
• RQ5 Can the programming model remain unchanged when integrating stochastic hardware accelerators into existing machine learning workflows?

Key findings

• Deep neural networks trained on stochastic hardware with N = 256 achieved a test error rate of 1.62%, slightly better than the control network’s 1.75% error rate using precise computation.
• Reducing the stochastic bit-sequence length N led to a monotonic increase in training cross-entropy error, but test accuracy remained stable or improved across all N values tested.
• The observed improvement in classification accuracy when training on noisy hardware suggests that the stochastic noise may act as a weak regularizer, enhancing generalization.
• Networks trained with weight pre-training via stacked sparse autoencoders on stochastic hardware achieved a test error of 1.35% for the control model, with stochastic hardware training yielding even lower error rates.
• A wide range of N values (from 64 to 512) produced comparable or better performance than precise computation, indicating a broad tolerance window for hardware noise.
• The results demonstrate that stochastic hardware can be used as a viable accelerator without requiring changes to the software stack or model architecture.