[Paper Review] Accelerating bandwidth-bound deep learning inference with main-memory accelerators

Benjamin Y. Cho, Jeageun Jung+1 more|arXiv (Cornell University)|Nov 14, 2021

Parallel Computing and Optimization Techniques21 citations

TL;DR

This paper proposes StepStone, a main-memory accelerator architecture that accelerates bandwidth-bound deep learning inference workloads by executing GEMM operations directly in CPU main memory using processing-in-memory (PIM) units. By exploiting GEMM data locality and designing custom address-generation logic, StepStone achieves up to 16X speedup over a fast CPU and 2.4X speedup over prior main-memory acceleration techniques, with 12X lower minimum latency than a CPU on large, tall/skinny matrices.

ABSTRACT

Matrix-matrix multiplication operations (GEMMs) are important in many HPC and machine-learning applications. They are often mapped to discrete accelerators (e.g., GPUs) to improve performance. However, we find that large tall/skinny and fat/short matrices benefit little from discrete acceleration and also do not perform well on a CPU. Such matrices are prevalent in important workloads, such as deep-learning inference within large-scale datacenters. We demonstrate the large potential of accelerating these GEMMs with processing in the main CPU memory, where processing in memory units (PIMs) take advantage of otherwise untapped bandwidth without requiring data copies. We develop a novel GEMM execution flow and corresponding memory-side address-generation logic that exploits GEMM locality and enables long-running PIM kernels despite the complex address-mapping functions employed by the CPU. Our evaluation of StepStone variants at the channel, device, and within-device PIM levels demonstrate 12X better minimum latency than a CPU and 2.8X greater throughput for strict query latency constraints. End-to-end performance analysis of recent recommendation and language models shows that StepStone outperforms a fast CPU by up to 16X and also the best prior main-memory acceleration approaches by up to 2.4X.

Motivation & Objective

• To address the poor performance of large, tall/skinny and fat/short GEMM operations in deep learning inference on both CPUs and discrete accelerators.
• To explore the potential of processing-in-memory (PIM) for accelerating bandwidth-bound GEMM workloads that are poorly served by traditional acceleration.
• To design a novel GEMM execution flow and memory-side address-generation logic that enables long-running PIM kernels despite complex CPU address mappings.
• To evaluate StepStone across multiple PIM deployment levels—channel, device, and within-device—and demonstrate end-to-end performance gains in real-world models.

Proposed method

• Design a novel GEMM execution flow that exploits data locality in large, tall/skinny and fat/short matrices common in deep learning inference.
• Develop custom memory-side address-generation logic to handle complex CPU address-mapping functions while enabling long-running PIM kernels.
• Integrate PIM units directly into the main memory subsystem to eliminate data movement and exploit untapped memory bandwidth.
• Evaluate StepStone at three levels of granularity: channel-level, device-level, and within-device-level PIM integration.
• Optimize the PIM kernel execution to minimize latency and maximize throughput under strict query latency constraints.
• Conduct end-to-end performance analysis on real-world recommendation and language models to validate system-level gains.

Experimental results

Research questions

• RQ1 Can processing-in-memory (PIM) architectures effectively accelerate bandwidth-bound GEMM operations in deep learning inference workloads?
• RQ2 How can PIM kernels be made long-running despite complex address-mapping functions used by modern CPUs?
• RQ3 What performance gains can be achieved by integrating PIM units into the main memory subsystem compared to CPUs and discrete accelerators?
• RQ4 How does StepStone’s design scale across different PIM deployment levels—channel, device, and within-device?
• RQ5 To what extent do real-world recommendation and language models benefit from main-memory GEMM acceleration?

Key findings

• StepStone achieves 12X lower minimum latency than a CPU on bandwidth-bound GEMM operations, demonstrating significant latency reduction.
• StepStone delivers 2.8X higher throughput than a CPU under strict query latency constraints, showing improved efficiency.
• End-to-end evaluation shows StepStone outperforms a fast CPU by up to 16X in real-world recommendation and language models.
• StepStone improves performance by up to 2.4X over the best prior main-memory acceleration approaches.
• The proposed address-generation logic successfully enables long-running PIM kernels despite complex CPU address mappings, unlocking PIM potential for irregular workloads.
• Evaluation across channel, device, and within-device PIM levels confirms consistent performance gains, validating the scalability of the design.