[Paper Review] DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference

Udit Gupta, Samuel Hsia+7 more|arXiv (Cornell University)|Jan 8, 2020

Stochastic Gradient Optimization Techniques55 references38 citations

TL;DR

The paper presents DeepRecInfra, an end-to-end infrastructure for at-scale neural recommendation inference, and DeepRecSched, a hill-climbing scheduler that optimizes per-request batch size and GPU offloading to maximize throughput under tail-latency targets.

ABSTRACT

Neural personalized recommendation is the corner-stone of a wide collection of cloud services and products, constituting significant compute demand of the cloud infrastructure. Thus, improving the execution efficiency of neural recommendation directly translates into infrastructure capacity saving. In this paper, we devise a novel end-to-end modeling infrastructure, DeepRecInfra, that adopts an algorithm and system co-design methodology to custom-design systems for recommendation use cases. Leveraging the insights from the recommendation characterization, a new dynamic scheduler, DeepRecSched, is proposed to maximize latency-bounded throughput by taking into account characteristics of inference query size and arrival patterns, recommendation model architectures, and underlying hardware systems. By doing so, system throughput is doubled across the eight industry-representative recommendation models. Finally, design, deployment, and evaluation in at-scale production datacenter shows over 30% latency reduction across a wide variety of recommendation models running on hundreds of machines.

Motivation & Objective

• Motivate the need for at-scale neural recommendation inference optimization in large datacenters.
• Propose an end-to-end infrastructure (DeepRecInfra) that reflects industry-representative models, workloads, and tail-latency targets.
• Develop a dynamic scheduler (DeepRecSched) that co-designs request- and batch-level parallelism with hardware accelerators.
• Demonstrate throughput gains and latency reductions across multiple models and hardware setups in production-scale settings.

Proposed method

• Characterize eight state-of-the-art recommendation models to capture model-level heterogeneity and bottlenecks.
• Develop DeepRecInfra to model industry workloads, query arrival patterns (Poisson), and query size distributions from production datacenters.
• Introduce DeepRecSched, a hill-climbing scheduler that tunes per-request batch size and GPU-offload thresholds to maximize QPS under tail-latency targets.
• Evaluate DeepRecSched on CPU and GPU (GTX-1080Ti) within DeepRecInfra across Broadwell and Skylake CPUs, comparing to a static baseline.
• Analyze how hardware heterogeneity (CPU SIMD width, cache hierarchy) affects the optimal balance of request- vs batch-level parallelism.
• Demonstrate that DeepRecSched achieves substantial throughput gains and improved power efficiency over baselines.

Experimental results

Research questions

• RQ1 How does at-scale recommendation inference differ from other DNN workloads in terms of model architecture, memory access, and input features?
• RQ2 Can an end-to-end infrastructure (DeepRecInfra) faithfully model real-world production recommendation workloads for at-scale inference?
• RQ3 Can a hill-climbing based scheduler (DeepRecSched) maximize tail-latency constrained throughput by adapting batch sizes and accelerator offloading across diverse models and hardware?
• RQ4 What are the throughput and energy efficiency benefits of using GPUs and CPU+GPU hybrids for at-scale recommendation inference under realistic query distributions?
• RQ5 How does hardware heterogeneity influence the optimal placement of work between per-request parallelism and batch-level parallelism?

Key findings

• DeepRecInfra enables modeling of eight industry-representative models, realistic tail-latency targets, and production-like query patterns.
• DeepRecSched doubles system throughput under strict latency targets and outperforms a static scheduler across all eight models.
• CPU vs GPU evaluation shows DeepRecSched-CPU yields 1.7–2.7x throughput gains, while DeepRecSched-GPU yields 4.0–5.8x gains depending on the latency target and model.
• GPU acceleration is most beneficial for larger queries, with an optimal GPU-offload threshold that varies by model and tail latency target.
• Offloading to GPUs improves throughput but with data-transfer overheads; optimal thresholds balance speedups against transfer costs.
• The optimal batch size and offload threshold depend on model architecture, tail-latency target, and hardware platform.