[Paper Review] Deep Learning Recommendation Model for Personalization and Recommendation Systems

Maxim Naumov, Dheevatsa Mudigere+22 more|arXiv (Cornell University)|May 31, 2019

Recommender Systems and Techniques23 references390 citations

TL;DR

DLRM presents a state-of-the-art neural network for personalized recommendations that handles sparse categorical features via embeddings and dense features via MLPs, combining them with explicit second-order interactions and a final MLP to predict event probabilities. It also introduces a mixed model/data parallelism strategy to scale embeddings and MLPs on multi-GPU hardware.

ABSTRACT

With the advent of deep learning, neural network-based recommendation models have emerged as an important tool for tackling personalization and recommendation tasks. These networks differ significantly from other deep learning networks due to their need to handle categorical features and are not well studied or understood. In this paper, we develop a state-of-the-art deep learning recommendation model (DLRM) and provide its implementation in both PyTorch and Caffe2 frameworks. In addition, we design a specialized parallelization scheme utilizing model parallelism on the embedding tables to mitigate memory constraints while exploiting data parallelism to scale-out compute from the fully-connected layers. We compare DLRM against existing recommendation models and characterize its performance on the Big Basin AI platform, demonstrating its usefulness as a benchmark for future algorithmic experimentation and system co-design.

Motivation & Objective

• Motivate the need for neural models in personalized recommendations and CTR prediction.
• Propose a unified architecture (embeddings for categorical features, bottom dense MLP for continuous features) that combines factorization-machine style interactions with an MLP head.
• Provide an open-source PyTorch and Caffe2 implementation of DLRM.
• Address memory and compute challenges through a hybrid model/data parallelism strategy for large embedding tables.

Proposed method

• Embed categorical features via large embedding tables to map sparse inputs to dense vectors.
• Process continuous features with a bottom MLP to obtain dense representations of matching length to embeddings.
• Compute explicit second-order interactions by dot-products between embedding vectors and dense features, potentially feeding them through an interaction-focused path.
• Concatenate interactions with processed dense features and pass through a top MLP to produce a probability via a sigmoid.
• Train and evaluate using standard loss (cross-entropy) with SGD/Adagrad optimizers, comparing against baseline models such as Deep and Cross Networks.
• Implement a hybrid parallelism approach: model parallelism for embeddings (memory-bound) and data parallelism for MLPs (compute-bound), with a custom embedding routing and all-to-all communication (but not natively supported by PyTorch/Caffe2).

Experimental results

Research questions

• RQ1 How can embedding-based representations of categorical features be effectively integrated with dense feature processing to improve CTR/prediction performance?
• RQ2 What is the impact of second-order interactions in DLRM compared to higher-order interactions in other models?
• RQ3 Can a mixed model/data parallelism strategy efficiently train very large embedding tables on multi-GPU hardware?
• RQ4 How does DLRM compare to existing models (e.g., DCN) on public CTR datasets in terms of accuracy and parameter efficiency?

Key findings

• DLRM achieves slightly higher training and validation accuracy than DCN on the Criteo Ad Kaggle dataset without extensive hyperparameter tuning.
• On a single-socket device profile with a large embedding table and substantial MLPs, embedding lookups dominate the runtime on CPU, while fully connected layers dominate on GPU.
• The model contains up to hundreds of millions of parameters (about 540M in the reported configuration) balancing embedding and MLP parameters.
• A specialized hybrid parallelism approach (model parallelism for embeddings and data parallelism for MLPs) enables scaling to multi-GPU hardware, with performance profiling showing substantial improvements and realistic timing benchmarks (CPU ~256s, GPU ~62s for a sample configuration).
• The authors provide open-source implementations in PyTorch and Caffe2 and evaluate on the Big Basin AI platform with eight Nvidia Tesla V100 GPUs.