[論文レビュー] Challenges and Research Directions for Large Language Model Inference Hardware

Xiaoyu Ma, David Patterson|arXiv (Cornell University)|Jan 8, 2026

Advanced Neural Network Applications被引用数 0

ひとこと要約

要するに: その論文は、LLM推論のボトルネックは計算性能よりもメモリとインターコネクトにあり、これらの制約を克服するための4つのアーキテクチャ研究機会を概説しており、データセンターAIとモバイル適用性について議論している。

ABSTRACT

Large Language Model (LLM) inference is hard. The autoregressive Decode phase of the underlying Transformer model makes LLM inference fundamentally different from training. Exacerbated by recent AI trends, the primary challenges are memory and interconnect rather than compute. To address these challenges, we highlight four architecture research opportunities: High Bandwidth Flash for 10X memory capacity with HBM-like bandwidth; Processing-Near-Memory and 3D memory-logic stacking for high memory bandwidth; and low-latency interconnect to speedup communication. While our focus is datacenter AI, we also review their applicability for mobile devices.

研究の動機と目的

• Identify the primary hardware bottlenecks in large language model (LLM) inference.
• Propose architecture research directions to increase memory capacity and bandwidth for LLM inference.
• Assess the applicability of proposed hardware approaches to datacenter AI and mobile devices.
• Highlight gaps and directions for future research in LLM inference hardware.

提案手法

• Review and synthesize architectural challenges in LLM inference.
• Highlight four key architectural opportunities: High Bandwidth Flash, Processing-Near-Memory, 3D memory-logic stacking, and low-latency interconnect.
• Discuss applicability to datacenter AI and mobile contexts.

実験結果

リサーチクエスチョン

• RQ1 What are the main hardware bottlenecks in LLM inference compared to training?
• RQ2 What architectural strategies can provide higher memory capacity and bandwidth for LLM inference?
• RQ3 How can near-memory processing and 3D memory-logic stacking reduce latency and improve throughput for LLM inference?
• RQ4 What role does interconnect latency play in LLM inference performance, and how can it be mitigated?
• RQ5 How applicable are the proposed hardware strategies to mobile devices versus datacenter AI deployments?

主な発見

• Identifies memory and interconnect as primary bottlenecks in LLM inference rather than compute.
• Places emphasis on four architectural directions to address these bottlenecks: High Bandwidth Flash, Processing-Near-Memory, 3D memory-logic stacking, and low-latency interconnect.
• Provides a discussion of the relevance of these approaches for datacenter AI and an assessment of mobility applicability.