[Paper Review] Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications

Jongsoo Park, Maxim Naumov+26 more|arXiv (Cornell University)|Nov 24, 2018

Parallel Computing and Optimization Techniques45 references82 citations

TL;DR

The paper characterizes Facebook’s DL inference workloads, analyzes compute/memory needs, and proposes CPU-focused optimizations and hardware considerations for future inference accelerators.

ABSTRACT

The application of deep learning techniques resulted in remarkable improvement of machine learning models. In this paper provides detailed characterizations of deep learning models used in many Facebook social network services. We present computational characteristics of our models, describe high performance optimizations targeting existing systems, point out their limitations and make suggestions for the future general-purpose/accelerated inference hardware. Also, we highlight the need for better co-design of algorithms, numerics and computing platforms to address the challenges of workloads often run in data centers.

Motivation & Objective

• Characterize DL inference workloads used in Facebook services and their evolution.
• Identify computational and memory access patterns that drive hardware design.
• Propose performance optimizations and reduced-precision techniques for CPU-based inference.
• Highlight hardware implications and requirements for future DL inference hardware.

Proposed method

• Classify inference workloads into ranking/recommendation, CV, and language tasks.
• Analyze arithmetic intensity, memory bandwidth, and on-chip/off-chip memory usage across representative models.
• Profile operator workloads and bottlenecks in CPU-based inference using an observer framework and roofline models.
• Develop reduced-precision DL inference techniques (fp16, int8) and outlier-aware quantization to improve throughput while maintaining accuracy.
• Propose software and interface considerations to better support DL workloads (convolution as first-class, non-square/memory-bound shapes).
• Present benchmark-driven insights and model/kernel characteristics to guide hardware-software co-design.

Experimental results

Research questions

• RQ1 What are the representative DL inference workloads in Facebook data centers and how have they evolved?
• RQ2 What are the compute and memory characteristics (arithmetic intensity, bandwidth needs, on-chip vs off-chip memory usage) of these workloads?
• RQ3 What optimizations (precision, kernel implementations, software design) can improve CPU-based inference performance within power/throughput constraints?
• RQ4 What hardware features and co-design strategies are needed for future DL inference hardware to handle these workloads effectively?

Key findings

• Embedding lookups dominate memory bandwidth and require large memory capacity, with embeddings often exceeding tens of GBs.
• On CPUs, FC layers and embedding lookups are major time sinks, and performance hinges on memory bandwidth for small batch sizes and group convolutions.
• Reduced-precision methods (fp16, int8 with 32-bit accumulation, outlier-aware quantization) can yield up to ~2–4x speedups in select layers while preserving accuracy within small margins.
• A roofline analysis shows performance gains depend on on-chip memory capacity and bandwidth; larger on-chip memory can significantly improve throughput for large activations and embeddings.
• Specialized kernels like FBGEMM for fp16 and i8-acc32 substantially outperform fp32 GEMM in representative models, particularly for FCs and Faster-RCNN-Shuffle.
• The paper argues for treating convolution as a first-class operation and for non-standard interfaces to accommodate tall-skinny matrices and group/depth-wise convolutions.