[Paper Review] FCN: Fusing Exponential and Linear Cross Network for Click-Through Rate Prediction

Honghao Li, Yiwen Zhang+4 more|arXiv (Cornell University)|Jul 18, 2024

Radiomics and Machine Learning in Medical Imaging5 citations

TL;DR

The paper introduces DCNv3 and SDCNv3, explicit feature interaction networks that exponentially grow feature cross orders with Self-Mask noise filtering, plus Tri-BCE supervision, achieving state-of-the-art results on six CTR datasets without relying on DNNs for implicit interactions.

ABSTRACT

As an important modeling paradigm in click-through rate (CTR) prediction, the Deep & Cross Network (DCN) and its derivative models have gained widespread recognition primarily due to their success in a trade-off between computational cost and performance. This paradigm employs a cross network to explicitly model feature interactions with linear growth, while leveraging deep neural networks (DNN) to implicitly capture higher-order feature interactions. However, these models still face several key limitations: (1) The performance of existing explicit feature interaction methods lags behind that of implicit DNN, resulting in overall model performance being dominated by the DNN; (2) While these models claim to capture high-order feature interactions, they often overlook potential noise within these interactions; (3) The learning process for different interaction network branches lacks appropriate supervision signals; and (4) The high-order feature interactions captured by these models are often implicit and non-interpretable due to their reliance on DNN. To address the identified limitations, this paper proposes a novel model, called Fusing Cross Network (FCN), along with two sub-networks: Linear Cross Network (LCN) and Exponential Cross Network (ECN). FCN explicitly captures feature interactions with both linear and exponential growth, eliminating the need to rely on implicit DNN. Moreover, we introduce the Self-Mask operation to filter noise layer by layer and reduce the number of parameters in the cross network by half. To effectively train these two cross networks, we propose a simple yet effective loss function called Tri-BCE, which provides tailored supervision signals for each network. We evaluate the effectiveness, efficiency, and interpretability of FCN on six benchmark datasets. Furthermore, by integrating LCN and ECN, FCN achieves a new state-of-the-art performance.

Motivation & Objective

• Motivate interpretable CTR models with explicit feature interactions beyond traditional DNN-based implicit interactions.
• Propose Deep Crossing (DCNv3) to exponentially grow cross orders for genuine deep crossing.
• Propose Shallow & Deep Cross Network v3 (SDCNv3) to fuse low- and high-order explicit interactions.
• Introduce Self-Mask to filter noise and reduce parameters in the cross network.
• Develop Tri-BCE loss to provide adaptive supervision signals to sub-networks.

Proposed method

• Embed and reshape multi-field categorical inputs into two views via chunking for shared Cross & Masked vectors.
• Define Deep Crossing (DCNv3) that grows cross orders exponentially using a cross vector and masked concatenation in each layer.
• Introduce Self-Mask: Mask(c_l) = c_l ⊙ max(0, LayerNorm(c_l)) to filter noise and halve parameters.
• Develop Shallow & Deep Cross Network v3 (SDCNv3) combining shallow and deep explicit crossing with a Self-Mask and a parallel fusion scheme.
• Propose Tri-BCE loss: L_Tri = L + w_D L_D + w_S L_S with adaptive weights w_D = max(0, L_D − L) and w_S = max(0, L_S − L).
• Provide a complexity comparison showing explicit-only DCNv3/SDCNv3 advantages in parameter and computation profiles.

Experimental results

Research questions

• RQ1 RQ1 Do DCNv3 and SDCNv3 outperform other CTR models in terms of performance on large-scale datasets?
• RQ2 RQ2 Are DCNv3 and SDCNv3 more efficient than competing CTR models?
• RQ3 RQ3 Does SDCNv3 offer interpretability and noise-filtering capabilities?
• RQ4 RQ4 How do different configurations affect model performance and training?

Key findings

• SDCNv3 achieves best performance across all six datasets and ranks first on PapersWithCode benchmarks for Criteo, KDD12, and KKBox.
• DCNv3 shows superior explicit interaction performance over strong baselines, with measurable improvements in Logloss and AUC on Avazu and Criteo.
• SDCNv3 provides state-of-the-art results using only explicit feature interactions, highlighting the effectiveness of explicit crossing when paired with Tri-BCE supervision.
• Tri-BCE provides adaptive supervision signals to sub-networks, leading to enhanced training dynamics and better final predictions.
• The models demonstrate favorable efficiency-complexity trade-offs, with Self-Mask reducing parameter counts and avoiding heavy implicit-DNN components.