[Paper Review] First-Generation Inference Accelerator Deployment at Facebook

Michael J. Anderson, Benny Chen+113 more|arXiv (Cornell University)|Jul 8, 2021

Distributed and Parallel Computing Systems51 references19 citations

TL;DR

Facebook co-designed a high-performance, energy-efficient inference accelerator system with six low-power cards (13W each) connected via PCIe, achieving 180–270 TOPS (int8) and 96 GB of LPDDR memory in a 91W power envelope. The open ecosystem, built on OCP hardware and open-source software (PyTorch, Caffe2, Glow), enables efficient deployment of large-scale ML models—including complex NLP and recommendation systems—outperforming CPUs in latency and throughput.

ABSTRACT

In this paper, we provide a deep dive into the deployment of inference accelerators at Facebook. Many of our ML workloads have unique characteristics, such as sparse memory accesses, large model sizes, as well as high compute, memory and network bandwidth requirements. We co-designed a high-performance, energy-efficient inference accelerator platform based on these requirements. We describe the inference accelerator platform ecosystem we developed and deployed at Facebook: both hardware, through Open Compute Platform (OCP), and software framework and tooling, through Pytorch/Caffe2/Glow. A characteristic of this ecosystem from the start is its openness to enable a variety of AI accelerators from different vendors. This platform, with six low-power accelerator cards alongside a single-socket host CPU, allows us to serve models of high complexity that cannot be easily or efficiently run on CPUs. We describe various performance optimizations, at both platform and accelerator level, which enables this platform to serve production traffic at Facebook. We also share deployment challenges, lessons learned during performance optimization, as well as provide guidance for future inference hardware co-design.

Motivation & Objective

• Address the growing demand for high-compute, low-latency inference in Facebook’s data centers due to increasingly complex ML models.
• Overcome limitations of CPU-based inference for large-scale models, especially those with sparse memory access, high model size, and bandwidth demands.
• Co-design a hardware-software stack that supports diverse ML workloads (NLP, CV, recommendation systems) with high performance and energy efficiency.
• Enable extensibility and interoperability by building an open ecosystem for multi-vendor AI accelerators and frameworks.
• Optimize the system across model, card, operator, and system levels to maximize utilization and minimize latency in production environments.

Proposed method

• Co-designed a six-card accelerator system using low-power, high-TOPS/W cards (30–45 TOPS/int8, 4–6 TFLOPS/FP16) connected via PCIe switch to scale performance and memory capacity.
• Deployed a multi-card partitioning strategy tailored for large recommendation system models, enabling efficient distribution of embedding tables and computation across cards.
• Implemented software stack optimizations across model (quantization, operator co-design), card (graph partitioning, batching, core placement), and system levels (serving stack modifications).
• Leveraged open-source frameworks (PyTorch, Caffe2, Glow) and tools (ONNXifi) to ensure portability and support for multiple accelerators and model types.
• Enabled programmability in vector cores to support evolving model compression techniques like 4-bit quantization and row-wise pruning.
• Explored future hardware extensions such as integrated video decoding and processing-in-memory (PIM) with programmability to adapt to future model and workload changes.

Experimental results

Research questions

• RQ1 How can a scalable, energy-efficient inference accelerator system be co-designed to meet the unique demands of large-scale, production ML workloads at Facebook?
• RQ2 What software and system-level optimizations are required to achieve high utilization and low latency when deploying complex models on multi-card accelerator platforms?
• RQ3 How can an open hardware and software ecosystem be architected to support interoperability across diverse AI accelerators and frameworks?
• RQ4 What role does programmability play in enabling support for evolving model compression and optimization techniques like 4-bit quantization and row-wise pruning?
• RQ5 What are the key challenges and opportunities in extending the accelerator system with specialized components such as video decoders or processing-in-memory (PIM)?

Key findings

• The six-card accelerator system delivers 180–270 TOPS (int8) and 24–36 TFLOPS (FP16) with 96 GB of LPDDR memory, achieving a peak efficiency of 2.0–3.0 TOPS/W.
• After software optimizations, the system efficiently serves large recommendation system models (up to 300× larger than current models) that are infeasible on CPUs due to memory and compute constraints.
• The open ecosystem—based on OCP hardware and open-source software (PyTorch, Caffe2, Glow, ONNXifi)—supports multiple vendors and frameworks, enabling broader innovation and portability.
• Model-level optimizations such as quantization and operator co-design significantly improved performance and reduced memory pressure on the accelerator cards.
• Programmable vector cores enabled runtime support for emerging compression techniques like 4-bit quantization and row-wise pruning, even after initial hardware design.
• The system demonstrated substantial improvements in throughput and latency over CPU-based inference, particularly for high-complexity models in NLP and recommendation systems.