[论文解读] TraNNsformer: Neural Network Transformation for memristive crossbar based neuromorphic system design

Aayush Ankit, Abhronil Sengupta+1 more|arXiv (Cornell University)|Nov 13, 2017

Advanced Memory and Neural Computing参考文献 28被引用 27

一句话总结

TraNNsformer 是一种技术感知的训练框架，可将脉冲神经网络转换为最优剪枝、聚类化的连接模式，与阻变交叉阵列（MCA）兼容，实现无精度损失的 28–55% 面积和 49–67% 能耗降低。它通过迭代剪枝连接、形成聚类并微调，使 DNN 与 MCA 的规则结构对齐，从而实现高效的硬件映射。

ABSTRACT

Implementation of Neuromorphic Systems using post Complementary Metal-Oxide-Semiconductor (CMOS) technology based Memristive Crossbar Array (MCA) has emerged as a promising solution to enable low-power acceleration of neural networks. However, the recent trend to design Deep Neural Networks (DNNs) for achieving human-like cognitive abilities poses significant challenges towards the scalable design of neuromorphic systems (due to the increase in computation/storage demands). Network pruning [7] is a powerful technique to remove redundant connections for designing optimally connected (maximally sparse) DNNs. However, such pruning techniques induce irregular connections that are incoherent to the crossbar structure. Eventually they produce DNNs with highly inefficient hardware realizations (in terms of area and energy). In this work, we propose TraNNsformer - an integrated training framework that transforms DNNs to enable their efficient realization on MCA-based systems. TraNNsformer first prunes the connectivity matrix while forming clusters with the remaining connections. Subsequently, it retrains the network to fine tune the connections and reinforce the clusters. This is done iteratively to transform the original connectivity into an optimally pruned and maximally clustered mapping. We evaluated the proposed framework by transforming different Multi-Layer Perceptron (MLP) based Spiking Neural Networks (SNNs) on a wide range of datasets (MNIST, SVHN and CIFAR10) and executing them on MCA-based systems to analyze the area and energy benefits. Without accuracy loss, TraNNsformer reduces the area (energy) consumption by 28% - 55% (49% - 67%) with respect to the original network. Compared to network pruning, TraNNsformer achieves 28% - 49% (15% - 29%) area (energy) savings. Furthermore, TraNNsformer is a technology-aware framework that allows mapping a given DNN to any MCA size permissible by the memristive technology for reliable operations.

研究动机与目标

• 解决不规则剪枝 DNN 与阻变交叉阵列（MCA）的规则交叉阵列架构之间的不兼容性。
• 降低因剪枝 DNN 中不规则连接性导致的硬件面积与能耗低效问题。
• 开发一种将 DNN 映射到任意 MCA 尺寸的框架，同时确保可靠运行。
• 通过使网络结构与 MCA 硬件约束对齐，实现高效、低功耗神经形态系统设计。

提出的方法

• 剪枝 DNN 的连接矩阵，以去除冗余连接，同时将剩余连接分组为聚类。
• 通过迭代再训练微调权重并强化聚类形成。
• 将原始 DNN 转换为与 MCA 交叉阵列结构兼容的最大稀疏化和聚类化连接模式。
• 在剪枝与再训练之间应用反馈回路，以优化聚类凝聚力与网络精度。
• 确保最终网络结构可扩展，并兼容由阻变技术允许的任意 MCA 尺寸。
• 采用技术感知方法，将 DNN 直接映射到基于 MCA 的神经形态硬件，实现最小面积与能耗开销。

实验结果

研究问题

• RQ1 如何将剪枝后的 DNN 转换为与阻变交叉阵列结构约束对齐的连接模式？
• RQ2 在基于 MCA 的系统中使用结构化连接时，可实现的最大面积与能耗降低幅度是多少？
• RQ3 所提出的框架是否能在实现高效硬件映射的同时保持网络精度？
• RQ4 与标准网络剪枝相比，该框架在硬件效率方面的表现如何？
• RQ5 该框架在不同 MCA 尺寸下可扩展到何种程度，同时保持可靠性与性能？

主要发现

• 与基于 MCA 的原始 DNN 相比，TraNNsformer 在 MCA 基础系统上实现了 28% 至 55% 的面积消耗降低和 49% 至 67% 的能耗降低。
• 与标准网络剪枝技术相比，该框架实现了 28% 至 49% 更高的面积节省和 15% 至 29% 更高的能耗节省。
• 在 MNIST、SVHN 和 CIFAR10 数据集上，转换后的网络保持了与原始 DNN 相同的精度。
• 该框架可将任意 DNN 映射到由底层阻变技术允许的任意 MCA 尺寸，确保可扩展性与可靠性。
• 迭代剪枝与再训练过程成功形成了最大聚类化、稀疏化的连接模式，与 MCA 硬件高度兼容。
• 该方法在多种数据集与网络架构上均表现出一致的性能提升，证实了其通用性。