[Paper Review] Operation-aware Neural Networks for User Response Prediction

Yi Yang, Baile Xu+2 more|arXiv (Cornell University)|Apr 2, 2019

Recommender Systems and Techniques12 references4 citations

TL;DR

This paper proposes Operation-aware Neural Networks (ONN), a deep learning model that learns distinct feature representations for different operations (e.g., inner-product, outer-product) in user response prediction. By decoupling embeddings per operation, ONN improves modeling of high-order feature interactions, achieving state-of-the-art performance on CTR/CVR prediction with faster convergence, especially in online training settings.

ABSTRACT

User response prediction makes a crucial contribution to the rapid development of online advertising system and recommendation system. The importance of learning feature interactions has been emphasized by many works. Many deep models are proposed to automatically learn high-order feature interactions. Since most features in advertising system and recommendation system are high-dimensional sparse features, deep models usually learn a low-dimensional distributed representation for each feature in the bottom layer. Besides traditional fully-connected architectures, some new operations, such as convolutional operations and product operations, are proposed to learn feature interactions better. In these models, the representation is shared among different operations. However, the best representation for different operations may be different. In this paper, we propose a new neural model named Operation-aware Neural Networks (ONN) which learns different representations for different operations. Our experimental results on two large-scale real-world ad click/conversion datasets demonstrate that ONN consistently outperforms the state-of-the-art models in both offline-training environment and online-training environment.

Motivation & Objective

• To address the limitation of shared feature representations across operations in existing deep models for user response prediction.
• To improve modeling of high-order feature interactions by learning operation-specific embeddings.
• To enhance performance in both offline and online training environments, particularly for real-time advertising systems.
• To enable more flexible and effective feature representation learning in neural architectures for CTR/CVR prediction.

Proposed method

• Proposes an operation-aware embedding layer that learns distinct low-dimensional representations for each feature depending on the operation it is used in.
• Introduces Operation-aware Neural Networks (ONN), which applies multiple operations (e.g., inner-product, outer-product) on operation-specific embeddings rather than shared ones.
• Employs a multi-layer perceptron (MLP) on the output of operations to learn nonlinear interactions, with sigmoid output for probability estimation.
• Supports multiple operation types, including inner-product, outer-product, and sub-network operations, with the option to combine them.
• Uses a shared embedding matrix per feature but applies different transformations per operation, enabling operation-specific representation learning.
• Applies the model to CTR/CVR prediction tasks using real-world ad click/conversion datasets for evaluation.

Experimental results

Research questions

• RQ1 Can learning operation-specific embeddings improve feature representation learning in user response prediction?
• RQ2 Does decoupling embeddings per operation lead to better performance in CTR/CVR prediction compared to shared representations?
• RQ3 How does ONN perform in online training environments compared to existing models?
• RQ4 Which operations (e.g., inner-product, outer-product) yield the best trade-off between performance and computational cost?
• RQ5 Can ONN generalize across different datasets and training regimes, including offline and online learning?

Key findings

• ONN consistently outperforms state-of-the-art models on both the Criteo and Tencent Ad datasets in terms of logloss, AUC, and RMSE.
• The combination of inner-product and outer-product operations achieves the best performance, with logloss of 0.43541 on Criteo and 0.10502 on Tencent Ad.
• ONN converges faster than baseline models and shows superior performance in online training environments, indicating strong suitability for real-time systems.
• The sub-network operation provides competitive results, achieving the highest AUC (0.81178) on the Criteo dataset, suggesting its potential for complex interaction modeling.
• The inner-product operation alone performs best overall in terms of balance between performance and efficiency, with the lowest computational cost.
• ONN achieved first prize in the Tencent Social Advertising College Algorithm Competition, validating its real-world effectiveness.