[Paper Review] ContextNet: A Click-Through Rate Prediction Framework Using Contextual information to Refine Feature Embedding

Zhiqiang Wang, Qingyun She+2 more|arXiv (Cornell University)|Jul 26, 2021

Recommender Systems and Techniques30 references4 citations

TL;DR

This paper proposes ContextNet, a novel CTR prediction framework that dynamically refines feature embeddings using contextual information from the input instance, enabling implicit modeling of high-order feature interactions. By integrating a contextual embedding module and a ContextNet block with non-linear mapping, ContextNet-PFFN and ContextNet-SFFN outperform SOTA models like DeepFM and xDeepFM on four real-world datasets, achieving state-of-the-art AUC performance.

ABSTRACT

Click-through rate (CTR) estimation is a fundamental task in personalized advertising and recommender systems and it's important for ranking models to effectively capture complex high-order features.Inspired by the success of ELMO and Bert in NLP field, which dynamically refine word embedding according to the context sentence information where the word appears, we think it's also important to dynamically refine each feature's embedding layer by layer according to the context information contained in input instance in CTR estimation tasks. We can effectively capture the useful feature interactions for each feature in this way. In this paper, We propose a novel CTR Framework named ContextNet that implicitly models high-order feature interactions by dynamically refining each feature's embedding according to the input context. Specifically, ContextNet consists of two key components: contextual embedding module and ContextNet block. Contextual embedding module aggregates contextual information for each feature from input instance and ContextNet block maintains each feature's embedding layer by layer and dynamically refines its representation by merging contextual high-order interaction information into feature embedding. To make the framework specific, we also propose two models(ContextNet-PFFN and ContextNet-SFFN) under this framework by introducing linear contextual embedding network and two non-linear mapping sub-network in ContextNet block. We conduct extensive experiments on four real-world datasets and the experiment results demonstrate that our proposed ContextNet-PFFN and ContextNet-SFFN model outperform state-of-the-art models such as DeepFM and xDeepFM significantly.

Motivation & Objective

• To address the challenge of effectively modeling high-order feature interactions in CTR prediction, which are critical for accurate ranking in recommender systems.
• To overcome the limitations of shallow MLPs in DNN-based models that fail to capture complex feature interactions efficiently.
• To develop a flexible, interpretable framework that enables dynamic, context-aware refinement of feature embeddings during inference.
• To design a scalable architecture that implicitly models high-order interactions without significantly increasing computational cost.

Proposed method

• ContextNet introduces a two-component framework: a contextual embedding module that aggregates context-aware information for each feature from the input instance.
• The ContextNet block maintains and dynamically refines each feature’s embedding layer by layer using a non-linear mapping sub-network that fuses high-order interaction signals.
• Two specific models, ContextNet-PFFN and ContextNet-SFFN, are proposed by integrating linear and non-linear contextual embedding networks and two distinct non-linear mapping sub-networks.
• The framework uses multiplicative interaction patterns between context embeddings and hidden states, inspired by the 'latent cross' mechanism, to model feature interactions more effectively than additive aggregation.
• The model employs a residual-like structure across blocks, enabling deep network training with improved stability and performance.
• Feature importance is analyzed via dot product matrices of embedding outputs across blocks, visualizing evolving feature correlations.

Experimental results

Research questions

• RQ1 Can dynamic refinement of feature embeddings using contextual information improve high-order feature interaction modeling in CTR prediction?
• RQ2 How does ContextNet’s architecture compare to existing models like DeepFM and xDeepFM in terms of AUC and computational efficiency?
• RQ3 To what extent does the number of ContextNet blocks and embedding size affect model performance and generalization?
• RQ4 Does the dynamic embedding mechanism effectively capture and adapt to relevant feature interactions per input instance?

Key findings

• ContextNet-PFFN and ContextNet-SFFN achieve state-of-the-art AUC performance on four real-world datasets, outperforming DeepFM and xDeepFM.
• On the Criteo dataset, ContextNet-PFFN achieves an AUC of 0.8113, which is higher than the 0.8104 baseline with embedding size 10.
• Model performance peaks at an embedding size of 50, with AUC of 0.8113 on Criteo and 0.7417 on Malware, before degrading due to overfitting at larger sizes.
• The number of ContextNet blocks significantly impacts performance, with optimal results observed at 10–11 blocks, indicating good trainability in deep architectures.
• Dynamic embedding refinement is empirically validated: feature correlations evolve across blocks, with 'age', 'movie genres', and 'movie year' emerging as key correlated features in a positive-prediction instance.
• Feature importance analysis shows 'age'=35 as the most influential feature in a sampled ML-1m instance, aligning with the model’s dynamic attention to contextually relevant features.