[Paper Review] DeepFM: A Factorization-Machine based Neural Network for CTR Prediction

Huifeng Guo, Ruiming Tang+3 more|arXiv (Cornell University)|Mar 13, 2017

Recommender Systems and TechniquesComputer Science18 references540 citations

TL;DR

DeepFM merges a factorization-machine component with a deep neural network in a shared-input architecture to model both low- and high-order feature interactions for CTR prediction, without feature engineering or pre-training.

ABSTRACT

Learning sophisticated feature interactions behind user behaviors is critical in maximizing CTR for recommender systems. Despite great progress, existing methods seem to have a strong bias towards low- or high-order interactions, or require expertise feature engineering. In this paper, we show that it is possible to derive an end-to-end learning model that emphasizes both low- and high-order feature interactions. The proposed model, DeepFM, combines the power of factorization machines for recommendation and deep learning for feature learning in a new neural network architecture. Compared to the latest Wide \& Deep model from Google, DeepFM has a shared input to its "wide" and "deep" parts, with no need of feature engineering besides raw features. Comprehensive experiments are conducted to demonstrate the effectiveness and efficiency of DeepFM over the existing models for CTR prediction, on both benchmark data and commercial data.

Motivation & Objective

• Motivate the need to capture both low- and high-order feature interactions in CTR prediction.
• Propose a unified neural architecture that combines FM and deep networks with shared input embeddings.
• Eliminate feature engineering and pre-training by enabling end-to-end training of the combined model.
• Evaluate DeepFM against state-of-the-art baselines on benchmark and industrial datasets.
• Demonstrate efficiency comparable to top deep models while improving predictive performance.

Proposed method

• Introduce a two-component DeepFM model: an FM component for order-2 interactions and a DNN component for high-order interactions.
• Share the same input feature embeddings between FM and DNN, enabling joint end-to-end training without pre-training.
• FM output: y_FM = <w,x> + sum_{i<j} <V_i,V_j> x_i x_j.
• Deep component: embed each field to a k-dimensional vector; a^(0) = [e_1,...,e_m]; a^(l+1) = sigma(W^(l) a^(l) + b^(l)); y_DNN = sigma(W^{H+1} a^(H) + b^{H+1}).
• Final prediction: y_hat = sigmoid(y_FM + y_DNN).
• Training jointly optimizes both FM and DNN parts, sharing embeddings, for end-to-end CTR prediction.
• Compared to Wide & Deep, DeepFM avoids feature engineering and pre-training, sharing embeddings across components.

Experimental results

Research questions

• RQ1 Can a single end-to-end model capture both low-order and high-order feature interactions for CTR prediction without feature engineering?
• RQ2 Does sharing embeddings between FM and DNN improve CTR prediction performance versus separate embeddings or pre-trained components?
• RQ3 Is the proposed DeepFM approach more efficient and effective than existing models (FM, FNN, IPNN, OPNN, PNN*, Wide & Deep) on benchmark and industrial data?
• RQ4 How does DeepFM perform on large-scale, real-world datasets in terms of AUC and LogLoss?
• RQ5 What are the practical considerations (training time, architecture choices) for deploying DeepFM in production?

Key findings

• DeepFM outperforms baselines on both benchmark and industrial data in AUC and LogLoss.
• On Company*, DeepFM achieves 0.8715 AUC and 0.02618 LogLoss, outperforming LR, FM, FNN, IPNN, OPNN, PNN*, LR&DNN, and FM&DNN variants.
• On Criteo, DeepFM achieves 0.8007 AUC and 0.45083 LogLoss, also superior to competing models.
• DeepFM provides efficiency close to the best deep models; pre-training is not required.
• Sharing embeddings between FM and DNN yields performance gains over models that use separate embeddings or pre-training.
• Overall, learning both low- and high-order interactions jointly with shared embeddings yields the best CTR prediction performance among evaluated models.