[Paper Review] Product-based Neural Networks for User Response Prediction over Multi-field Categorical Data

Yanru Qu, Bohui Fang+6 more|arXiv (Cornell University)|Jul 1, 2018

Recommender Systems and Techniques47 references22 citations

TL;DR

This paper proposes Product-based Neural Networks (PNNs) and a novel PIN (Product-network In Network) architecture to improve user response prediction on multi-field categorical data. By introducing kernel product layers to model field-aware feature interactions and addressing gradient insensitivity in deep networks, PNNs achieve state-of-the-art performance, outperforming 8 baselines on AUC and log loss across 4 industrial datasets and showing a 34.67% relative CTR improvement in online A/B testing.

ABSTRACT

User response prediction is a crucial component for personalized information retrieval and filtering scenarios, such as recommender system and web search. The data in user response prediction is mostly in a multi-field categorical format and transformed into sparse representations via one-hot encoding. Due to the sparsity problems in representation and optimization, most research focuses on feature engineering and shallow modeling. Recently, deep neural networks have attracted research attention on such a problem for their high capacity and end-to-end training scheme. In this paper, we study user response prediction in the scenario of click prediction. We first analyze a coupled gradient issue in latent vector-based models and propose kernel product to learn field-aware feature interactions. Then we discuss an insensitive gradient issue in DNN-based models and propose Product-based Neural Network (PNN) which adopts a feature extractor to explore feature interactions. Generalizing the kernel product to a net-in-net architecture, we further propose Product-network In Network (PIN) which can generalize previous models. Extensive experiments on 4 industrial datasets and 1 contest dataset demonstrate that our models consistently outperform 8 baselines on both AUC and log loss. Besides, PIN makes great CTR improvement (relatively 34.67%) in online A/B test.

Motivation & Objective

• To address the coupled gradient issue in latent vector-based models like FFM, which limits effective field-aware feature interaction learning.
• To overcome the insensitive gradient problem in standard DNNs when training on sparse, one-hot encoded multi-field categorical data.
• To design a deep learning architecture that explicitly models high-order feature interactions while maintaining efficiency and scalability.
• To develop a generalizable framework that unifies and improves upon existing models like FM, FFM, and standard DNNs for click-through rate prediction.
• To validate the proposed models on real-world industrial datasets and demonstrate superior performance in both offline metrics and online A/B testing.

Proposed method

• Proposes a kernel product layer that computes field-aware feature interactions by applying a learnable kernel to pairwise interactions between field-specific embeddings.
• Introduces the Product-based Neural Network (PNN) architecture, which combines a feature extractor with a product layer to model feature interactions before passing to a deep classifier.
• Develops the Product-network In Network (PIN) as a generalization of PNN, allowing the product layer to be replaced by a sub-network for more flexible interaction modeling.
• Employs a theoretical analysis to show that standard DNNs suffer from gradient insensitivity when learning poly-2 functions, which are common in user response prediction.
• Uses synthetic experiments with a poly-2 function to validate that DNNs fail to converge to the optimal solution despite universal approximation, highlighting the need for explicit interaction modeling.
• Trains models end-to-end using stochastic gradient descent, with AUC and log loss as evaluation metrics on both synthetic and real-world datasets.

Experimental results

Research questions

• RQ1 Can field-aware feature interactions be learned more effectively than in traditional latent vector models like FFM, which suffer from coupled gradients?
• RQ2 Does the inclusion of explicit product layers in deep networks improve performance on sparse, multi-field categorical data compared to standard DNNs?
• RQ3 Can a general architecture like PIN unify and outperform existing models such as FM, FFM, and standard DNNs in user response prediction?
• RQ4 To what extent does gradient insensitivity in DNNs hinder learning of simple, structured functions like poly-2 in sparse data settings?
• RQ5 Does the proposed model achieve measurable improvements in both offline metrics and online A/B testing in real-world industrial applications?

Key findings

• PNNs consistently outperform 8 baselines on both AUC and log loss across 4 industrial datasets, demonstrating superior generalization and predictive accuracy.
• The PIN architecture achieves a relative 34.67% improvement in CTR in online A/B testing, validating its real-world effectiveness.
• Synthetic experiments show that standard DNNs fail to converge to the optimal solution for a poly-2 function, even with increased depth and width, indicating inherent gradient insensitivity.
• The kernel product layer effectively captures field-aware feature interactions and mitigates the coupled gradient issue present in FFM and similar models.
• The proposed models show robust performance across diverse data sparsity levels, with training stability and convergence improvements over standard DNNs.
• The generalization capability of PIN is demonstrated by its ability to unify and improve upon existing models, suggesting a promising direction for future deep learning in information retrieval systems.