[Paper Review] CTRL: Connect Collaborative and Language Model for CTR Prediction

Xiang‐Yang Li, Bo Chen+2 more|arXiv (Cornell University)|Jun 5, 2023

Recommender Systems and Techniques4 citations

TL;DR

This paper proposes CTRL, a novel framework that connects collaborative and pre-trained language models for CTR prediction by converting tabular data into textual form and performing cross-modal knowledge alignment. The method achieves state-of-the-art performance with low inference latency, outperforming SOTA models by 5% in online CTR and maintaining industrial efficiency through a lightweight collaborative model deployed after alignment.

ABSTRACT

Traditional click-through rate (CTR) prediction models convert the tabular data into one-hot vectors and leverage the collaborative relations among features for inferring the user's preference over items. This modeling paradigm discards essential semantic information. Though some works like P5 and CTR-BERT have explored the potential of using Pre-trained Language Models (PLMs) to extract semantic signals for CTR prediction, they are computationally expensive and suffer from low efficiency. Besides, the beneficial collaborative relations are not considered, hindering the recommendation performance. To solve these problems, in this paper, we propose a novel framework extbf{CTRL}, which is industrial-friendly and model-agnostic with superior inference efficiency. Specifically, the original tabular data is first converted into textual data. Both tabular data and converted textual data are regarded as two different modalities and are separately fed into the collaborative CTR model and pre-trained language model. A cross-modal knowledge alignment procedure is performed to fine-grained align and integrate the collaborative and semantic signals, and the lightweight collaborative model can be deployed online for efficient serving after fine-tuned with supervised signals. Experimental results on three public datasets show that CTRL outperforms the state-of-the-art (SOTA) CTR models significantly. Moreover, we further verify its effectiveness on a large-scale industrial recommender system.

Motivation & Objective

• To address the limitation of traditional CTR models that discard semantic information through one-hot encoding of tabular features.
• To integrate semantic signals from pre-trained language models (PLMs) with collaborative signals from tabular data for improved recommendation accuracy.
• To overcome the high inference cost of PLMs in industrial settings by deploying only a lightweight collaborative model after knowledge alignment.
• To develop a model-agnostic, industrial-friendly framework that maintains high efficiency without extensive engineering optimization.

Proposed method

• Convert raw tabular features into natural language text using human-designed prompts to enable PLM processing.
• Treat tabular data and textual data as two distinct modalities, feeding them into a collaborative CTR model and a pre-trained language model, respectively.
• Perform cross-modal knowledge alignment via contrastive learning to fine-grained align and integrate collaborative and semantic signals.
• Fine-tune the collaborative model with supervised signals from the aligned representations for downstream CTR prediction.
• Deploy only the lightweight collaborative model in online serving, ensuring low-latency inference.
• Evaluate prompt engineering strategies to optimize textual input quality for PLM performance.

Experimental results

Research questions

• RQ1 How can collaborative and semantic signals be effectively combined to improve CTR prediction beyond isolated modeling?
• RQ2 Can cross-modal knowledge alignment between tabular and textual representations enhance recommendation performance?
• RQ3 To what extent can a lightweight collaborative model achieve SOTA performance after alignment with PLM signals?
• RQ4 Can the proposed framework maintain low inference latency while outperforming existing SOTA models in industrial settings?

Key findings

• CTRL achieves an AUC of 0.6683 on public datasets, significantly outperforming the best baseline (DeepFM at 0.6547) and SOTA models like CTR-BERT and P5.
• CTRL reduces logloss to 0.1606, demonstrating superior optimization performance compared to the next best model (AutoInt at 0.1713).
• In online A/B testing on a large-scale industrial system, CTRL achieved a 5% relative increase in CTR compared to the base model.
• The framework maintains the same inference latency as the backbone collaborative model, proving its industrial efficiency and deployability.
• Prompt engineering shows that natural language prompts (e.g., "User age: 25, gender: male") outperform rule-based or masked variants, with Prompt-1 yielding the best performance.
• The ablation study confirms that including feature fields (e.g., "age", "gender") and fluent, semantically rich sentences significantly improves model performance.