[Paper Review] Efficient ConvNets for Analog Arrays

Malte J. Rasch, Tayfun Gokmen+2 more|arXiv (Cornell University)|Jul 3, 2018

Advanced Memory and Neural Computing17 references3 citations

TL;DR

This paper proposes RAPA ConvNets, which replicate convolutional kernel matrices across multiple analog arrays and distribute computation randomly among them to accelerate training on mixed analog-digital hardware. Despite increased parameters, the method achieves comparable or better accuracy and robustness, with training speedups up to 128× and a 64% reduction in gap to perfect adversarial robustness on CIFAR-10.

ABSTRACT

Analog arrays are a promising upcoming hardware technology with the potential to drastically speed up deep learning. Their main advantage is that they compute matrix-vector products in constant time, irrespective of the size of the matrix. However, early convolution layers in ConvNets map very unfavorably onto analog arrays, because kernel matrices are typically small and the constant time operation needs to be sequentially iterated a large number of times, reducing the speed up advantage for ConvNets. Here, we propose to replicate the kernel matrix of a convolution layer on distinct analog arrays, and randomly divide parts of the compute among them, so that multiple kernel matrices are trained in parallel. With this modification, analog arrays execute ConvNets with an acceleration factor that is proportional to the number of kernel matrices used per layer (here tested 16-128). Despite having more free parameters, we show analytically and in numerical experiments that this convolution architecture is self-regularizing and implicitly learns similar filters across arrays. We also report superior performance on a number of datasets and increased robustness to adversarial attacks. Our investigation suggests to revise the notion that mixed analog-digital hardware is not suitable for ConvNets.

Motivation & Objective

• To address the poor mapping of early ConvNet layers onto analog arrays, which suffer from high sequential computation despite constant-time matrix operations.
• To enable efficient training of ConvNets on emerging mixed analog-digital hardware by reducing effective computation time per layer.
• To investigate whether model parallelism via kernel replication can maintain or improve performance while enabling hardware acceleration.
• To explore the implicit regularization and adversarial robustness benefits of random tiling in replicated kernel architectures.

Proposed method

• Replicate each convolutional kernel matrix across $n_t$ separate analog arrays (tiles), distributing input patches randomly among them.
• Train each tile independently with its own set of parameters, enabling parallel execution on analog arrays.
• Use majority voting across tiles during inference to aggregate predictions and improve robustness.
• Formulate the tiling strategy as a random assignment of image patches to tiles, avoiding explicit weight sharing.
• Analyze the implicit regularization effect via a simplified analytical model showing convergence toward similar filters across tiles.
• Simulate the RAPA ConvNet on conventional hardware to validate performance and robustness across datasets.

Experimental results

Research questions

• RQ1 Can model parallelism via kernel replication improve the efficiency of ConvNets on analog arrays, despite their small kernel sizes?
• RQ2 Does random assignment of computation across replicated kernel matrices lead to implicit regularization and stable convergence?
• RQ3 To what extent does RAPA ConvNet improve robustness against white-box adversarial attacks compared to standard ConvNets?
• RQ4 Can the RAPA architecture maintain or exceed the accuracy of conventional ConvNets while enabling significant speedups on analog hardware?
• RQ5 How does increasing the number of tiles affect training stability, generalization, and adversarial robustness?

Key findings

• The RAPA ConvNet achieves a test error of 19% on CIFAR-10 after 85 epochs with mixed pooling, comparable to the original network’s 82 epochs.
• With $n_t = (128,32,8)$, the network reaches 83.97% robust accuracy on CIFAR-10 for $ε \approx 33$, reducing the gap to perfect robustness by 64%.
• The RAPA architecture maintains performance with up to 20× more convolutional parameters than the original network, showing no overfitting (training error 15%).
• The method enables a theoretical 128× speedup per training epoch on analog arrays, assuming full parallelism across tiles.
• Random tiling acts as a confidence stabilization mechanism, reducing overconfident predictions and improving adversarial robustness.
• The architecture is robust to hyperparameter changes, with stable training and performance even at high tile counts, suggesting scalability to larger networks.