[Paper Review] Wide & Deep Learning for Recommender Systems

Heng-Tze Cheng, Levent Koç+14 more|arXiv (Cornell University)|Jun 24, 2016

Recommender Systems and Techniques8 references260 citations

TL;DR

This paper proposes Wide & Deep learning, jointly training a wide linear model with cross-product features and a deep neural network with embeddings to improve recommender systems, and demonstrates significant online gains on Google Play.

ABSTRACT

Generalized linear models with nonlinear feature transformations are widely used for large-scale regression and classification problems with sparse inputs. Memorization of feature interactions through a wide set of cross-product feature transformations are effective and interpretable, while generalization requires more feature engineering effort. With less feature engineering, deep neural networks can generalize better to unseen feature combinations through low-dimensional dense embeddings learned for the sparse features. However, deep neural networks with embeddings can over-generalize and recommend less relevant items when the user-item interactions are sparse and high-rank. In this paper, we present Wide & Deep learning---jointly trained wide linear models and deep neural networks---to combine the benefits of memorization and generalization for recommender systems. We productionized and evaluated the system on Google Play, a commercial mobile app store with over one billion active users and over one million apps. Online experiment results show that Wide & Deep significantly increased app acquisitions compared with wide-only and deep-only models. We have also open-sourced our implementation in TensorFlow.

Motivation & Objective

• Motivate the need to balance memorization and generalization in large-scale recommender systems with sparse inputs.
• Introduce the Wide & Deep learning framework that jointly trains wide and deep components.
• Describe productionized deployment on Google Play and evaluate online performance.
• Provide an open-source TensorFlow implementation and practical training/serving considerations.

Proposed method

• Define the wide component as a generalized linear model with cross-product feature transformations.
• Define the deep component as a feed-forward neural network with embedding layers for categorical features.
• Jointly train wide and deep components by combining their outputs and optimizing with a shared logistic loss.
• Use FTRL with L1 regularization for the wide part and AdaGrad for the deep part during training.
• Implement a warm-starting system to reuse embeddings and weights when retraining on new data.
• Optimize serving with multithreading and batching to meet low latency (≈10 ms) requirements.

Experimental results

Research questions

• RQ1 Can a hybrid Wide & Deep model improve recommendation accuracy over wide-only and deep-only models in a large-scale, sparse-input setting?
• RQ2 Does joint training of wide and deep components yield better online performance (acquisitions) and acceptable offline metrics compared to baselines?
• RQ3 What are the practical training and serving considerations (latency, throughput) for production deployment of such a hybrid model?

Key findings

• Wide & Deep achieved +3.9% online acquisition gain versus the wide-only control.
• Wide & Deep outperformed the deep-only model with an additional +1.0% online gain (statistically significant).
• Offline AUC for Wide & Deep was 0.728, slightly higher than deep (0.722) and wide (0.726).
• Serving latency was reduced to 14 ms per batch with multi-threading, enabling scalable deployment at Google Play scale.