[Paper Review] ALPINE: Analog In-Memory Acceleration with Tight Processor Integration for Deep Learning

Joshua Klein, Irem Boybat+7 more|arXiv (Cornell University)|May 20, 2022

Advanced Memory and Neural Computing4 citations

TL;DR

This paper presents ALPINE, a full-system simulation framework that enables tight integration of analog in-memory computing (AIMC) accelerators with multi-core ARM CPUs for deep learning inference. By extending gem5 with custom ARMv8 instructions and a software library (AIMCLib), ALPINE achieves up to 20.5x speedup and 20.8x energy reduction for CNNs, MLPs, and LSTMs compared to a SIMD-enabled digital CPU, demonstrating that hardware-software co-design can unlock high performance and efficiency in AI workloads without sacrificing flexibility.

ABSTRACT

Analog in-memory computing (AIMC) cores offers significant performance and energy benefits for neural network inference with respect to digital logic (e.g., CPUs). AIMCs accelerate matrix-vector multiplications, which dominate these applications' run-time. However, AIMC-centric platforms lack the flexibility of general-purpose systems, as they often have hard-coded data flows and can only support a limited set of processing functions. With the goal of bridging this gap in flexibility, we present a novel system architecture that tightly integrates analog in-memory computing accelerators into multi-core CPUs in general-purpose systems. We developed a powerful gem5-based full system-level simulation framework into the gem5-X simulator, ALPINE, which enables an in-depth characterization of the proposed architecture. ALPINE allows the simulation of the entire computer architecture stack from major hardware components to their interactions with the Linux OS. Within ALPINE, we have defined a custom ISA extension and a software library to facilitate the deployment of inference models. We showcase and analyze a variety of mappings of different neural network types, and demonstrate up to 20.5x/20.8x performance/energy gains with respect to a SIMD-enabled ARM CPU implementation for convolutional neural networks, multi-layer perceptrons, and recurrent neural networks.

Motivation & Objective

• To address the inflexibility of standalone analog in-memory computing (AIMC) accelerators that lack support for diverse neural network types, dynamic data flows, and varying precision formats.
• To bridge the gap between the high performance of AIMC and the programmability of general-purpose CPUs by enabling tight hardware-software co-design.
• To develop a validated, full-stack simulation framework capable of modeling complex interactions between AIMC tiles, CPUs, memory, and the Linux OS for accurate system-level evaluation.
• To enable efficient deployment of diverse DNNs—such as CNNs, MLPs, and LSTMs—on hybrid CPU-AIMC architectures through lightweight ISA extensions and a dedicated software library.
• To demonstrate that tightly coupled AIMC integration can achieve significant performance and energy gains without compromising system flexibility or requiring dedicated accelerators for each new model.

Proposed method

• Extended the gem5-X simulator with a full-system simulation framework to model ARMv8-based systems augmented with multiple AIMC tiles as tightly coupled accelerators.
• Designed and implemented custom ARMv8 64-bit instruction set extensions to directly control AIMC tile operations, enabling CPU-driven activation and data transfer.
• Developed a software library (AIMCLib) to abstract low-level AIMC operations, simplifying the deployment of DNN inference workloads on the hybrid architecture.
• Mapped various DNNs—CNNs (fast/medium/slow variants), MLPs, and LSTMs—onto the hybrid system, using fine-grained pipelining and selective acceleration of convolutional layers.
• Enabled dataflow optimization by flattening kernels and feature maps, and storing them in AIMC tile columns for efficient MAC operations in parallel.
• Conducted system-level simulations across low-power and high-power configurations to evaluate performance, energy, and CPU utilization under realistic workloads.

Experimental results

Research questions

• RQ1 Can tight integration of analog in-memory computing (AIMC) tiles with general-purpose CPUs achieve significant performance and energy gains for diverse DNN workloads?
• RQ2 How does the addition of custom ISA extensions and a software library impact the programmability and deployment efficiency of AIMC accelerators in a full-system simulation environment?
• RQ3 To what extent can AIMC acceleration reduce data movement and improve compute efficiency in CNNs, MLPs, and LSTMs compared to fully digital SIMD-optimized CPU implementations?
• RQ4 What are the performance and energy trade-offs when accelerating only specific layers (e.g., convolutional layers) in DNNs using AIMC, while keeping others on the CPU?
• RQ5 How does the system handle uneven CPU core utilization and communication bottlenecks in pipelined DNN inference workloads?

Key findings

• The ALPINE framework successfully enables full-system simulation of hybrid CPU-AIMC architectures, including interactions between the OS, memory, and custom hardware components.
• For the largest CNN variant (CNN-S), the system achieved a 20.5x speedup and 20.8x energy reduction compared to a SIMD-enabled ARM CPU implementation.
• Convolutional layers 2 and 3 saw up to 4x reduction in CPU idle cycles and up to 3x increase in instructions per cycle (IPC) when accelerated by AIMC tiles.
• The fully-connected layers exhibited the highest CPU idle time, indicating that bottlenecks shift across layers and require further optimization of data flow and memory access patterns.
• The performance gains were consistent across different DNN types, including MLPs and LSTMs, demonstrating the framework’s versatility in supporting diverse neural network workloads.
• The results validate that tight integration of AIMC with CPUs, enabled by custom instructions and software abstractions, unlocks high efficiency without sacrificing flexibility or requiring model-specific accelerator designs.