[Paper Review] Training large-scale ANNs on simulated resistive crossbar arrays

Malte J. Rasch, Tayfun Gokmen+1 more|arXiv (Cornell University)|Jun 6, 2019

Advanced Memory and Neural Computing18 references4 citations

TL;DR

This paper demonstrates that with algorithmic compensation—specifically normalization and weight range remapping—large-scale ANNs like AlexNet can be successfully trained on simulated resistive crossbar arrays despite hardware constraints such as limited weight resolution and analog noise. The key contribution is a 20-percentage-point improvement in top-1 test error (to ~60%) using only 12,000 effective device states, showing that these techniques dramatically close the accuracy gap with floating-point training.

ABSTRACT

Accelerating training of artificial neural networks (ANN) with analog resistive crossbar arrays is a promising idea. While the concept has been verified on very small ANNs and toy data sets (such as MNIST), more realistically sized ANNs and datasets have not yet been tackled. However, it is to be expected that device materials and hardware design constraints, such as noisy computations, finite number of resistive states of the device materials, saturating weight and activation ranges, and limited precision of analog-to-digital converters, will cause significant challenges to the successful training of state-of-the-art ANNs. By using analog hardware aware ANN training simulations, we here explore a number of simple algorithmic compensatory measures to cope with analog noise and limited weight and output ranges and resolutions, that dramatically improve the simulated training performances on RPU arrays on intermediately to large-scale ANNs.

Motivation & Objective

• To investigate the feasibility of training large-scale ANNs, such as AlexNet on ImageNet, using simulated resistive processing unit (RPU) arrays with hardware constraints.
• To evaluate the impact of analog noise, limited weight resolution, and restricted dynamic range on training performance.
• To identify and implement simple algorithmic compensatory measures that improve training accuracy under these constraints.
• To determine the minimum number of effective device states required to approach floating-point accuracy in large-scale training.

Proposed method

• Simulated training of large ANNs (e.g., AlexNet) on resistive crossbar arrays using a Caffe2-integrated RPU simulator with GPU acceleration.
• Employed pulsed weight updates with stochastic pulse trains (max length 31) to emulate analog hardware updates.
• Applied z-score normalization of activations to mitigate analog noise during forward and backward passes.
• Implemented a novel weight remapping scheme to virtually expand the effective number of resistive states and improve regularization.
• Used a baseline RPU model with 1,200 effective states (Δw_min = 0.001) and scaled up to 12,000 states for ablation.
• Evaluated performance on ImageNet with batch size 25, weight decay λ = 0.00125, and learning rate decay every 15 epochs.

Experimental results

Research questions

• RQ1 How many effective resistive states are required to train large-scale ANNs like AlexNet on ImageNet with acceptable accuracy on analog RPU arrays?
• RQ2 Can normalization techniques effectively reduce the impact of analog noise during backpropagation in RPU-based training?
• RQ3 Does virtual remapping of weight ranges improve the effective resolution and generalization performance on low-precision RPU arrays?
• RQ4 To what extent can algorithmic compensation close the accuracy gap between analog RPU training and floating-point reference models?
• RQ5 Is the performance gain from normalization and remapping sufficient to achieve state-of-the-art accuracy with current hardware constraints?

Key findings

• With only 1,200 effective device states, the top-1 test error on ImageNet reached approximately 80%, indicating significant performance degradation due to limited resolution.
• Applying z-score normalization of activations reduced the impact of analog noise and improved training stability, enabling convergence on larger models.
• Introducing weight range remapping reduced the top-1 test error by about 20 percentage points, achieving ~60% error with 12,000 effective states.
• Even with 12,000 states and full algorithmic compensation, the model failed to reach the floating-point reference accuracy of below 50% top-1 error.
• The performance gain from remapping was more pronounced in deeper networks like AlexNet (max matrix dimension 9216) than in shallower ones like ResNet on CIFAR.
• The results suggest that while algorithmic compensation significantly improves analog training, reaching full floating-point accuracy requires further innovations in materials or system design.