[论文解读] Addressing Delayed Feedback for Continuous Training with Neural Networks in CTR prediction

Sofia Ira Ktena, Alykhan Tejani+6 more|arXiv (Cornell University)|Jul 15, 2019

Data Stream Mining Techniques参考文献 27被引用 8

一句话总结

本文提出改进的损失函数——具体为FN加权和FN校准——以解决展示广告中点击率（CTR）预测神经网络持续训练时的延迟反馈问题。通过在训练过程中对未观测到的正样本标签的不确定性进行建模，这些方法减少了假阴性偏差，在在线实验中相较标准对数损失实现55%的千次请求收入（RPMq）提升。

ABSTRACT

One of the challenges in display advertising is that the distribution of features and click through rate (CTR) can exhibit large shifts over time due to seasonality, changes to ad campaigns and other factors. The predominant strategy to keep up with these shifts is to train predictive models continuously, on fresh data, in order to prevent them from becoming stale. However, in many ad systems positive labels are only observed after a possibly long and random delay. These delayed labels pose a challenge to data freshness in continuous training: fresh data may not have complete label information at the time they are ingested by the training algorithm. Naive strategies which consider any data point a negative example until a positive label becomes available tend to underestimate CTR, resulting in inferior user experience and suboptimal performance for advertisers. The focus of this paper is to identify the best combination of loss functions and models that enable large-scale learning from a continuous stream of data in the presence of delayed labels. In this work, we compare 5 different loss functions, 3 of them applied to this problem for the first time. We benchmark their performance in offline settings on both public and proprietary datasets in conjunction with shallow and deep model architectures. We also discuss the engineering cost associated with implementing each loss function in a production environment. Finally, we carried out online experiments with the top performing methods, in order to validate their performance in a continuous training scheme. While training on 668 million in-house data points offline, our proposed methods outperform previous state-of-the-art by 3% relative cross entropy (RCE). During online experiments, we observed 55% gain in revenue per thousand requests (RPMq) against naive log loss.

研究动机与目标

• 解决持续训练CTR预测模型时的延迟反馈挑战，其中正样本标签（点击）仅在随机延迟后被观测到。
• 在展示广告中常见的实际数据分布偏移背景下，评估并比较多种损失函数——尤其是新型损失函数。
• 识别在保持模型性能与可扩展性的同时，最小化假阴性标注的损失函数，适用于生产系统。
• 通过真实广告流量上的在线A/B实验验证最佳性能方法，衡量其对RPMq和货币化CTR等关键业务指标的影响。
• 为在流式训练流水线中以最低基础设施成本和低延迟实现这些损失函数提供工程指导。

提出的方法

• 提出五种损失函数，包括两种新型损失函数：FN加权损失和FN校准损失，用于处理训练期间未观测到的正样本标签。
• 应用重要性采样原理，根据延迟反馈的概率重新加权训练样本，减少虚假负样本带来的偏差。
• 采用两阶段训练策略：先在即时负样本上进行训练，随后通过损失重加权利用延迟正样本信号进行校正。
• 同时使用浅层模型（逻辑回归）和深层模型（wide & deep）神经网络架构，以评估在不同模型复杂度下的泛化能力。
• 实施在线训练，支持实时数据摄入与模型更新，使模型能够在不存储未标注数据的情况下适应分布偏移。
• 引入时间依赖加权机制，以考虑反馈的时间依赖性，提升梯度稳定性与收敛性。

实验结果

研究问题

• RQ1 在CTR预测模型的持续训练中，不同损失函数在处理延迟反馈时的表现如何？
• RQ2 哪种损失函数能在保持训练稳定性和高预测准确性的前提下，最小化假阴性偏差，且在离线与在线设置中表现优异？
• RQ3 新型损失函数如FN加权和FN校准是否能在RCE和RPMq等业务指标上超越标准对数损失与PU损失？
• RQ4 在在线环境中使用PU损失时，训练稳定性与性能之间的权衡如何？能否通过非负学习实现稳定化？
• RQ5 模型架构（如wide & deep与逻辑回归）与损失函数选择之间在性能与鲁棒性方面如何相互作用？

主要发现

• FN加权损失与FN校准损失在公开数据集和专有数据集上均取得了最佳离线性能，相较最先进方法将相对交叉熵（RCE）降低3%。
• 在在线实验中，FN加权损失与FN校准损失相较标准对数损失，分别使千次请求收入（RPMq）提升55.10%和54.37%。
• 货币化CTR分别提升23.01%和23.19%，表明其具有显著的业务影响。
• 尽管PU损失在离线设置中表现强劲，但在在线训练中两天内即出现发散，表明其在真实部署中存在不稳定性。
• 延迟反馈损失函数在简单模型上表现良好，但在深层模型中被FN校准与FN加权等更稳健的方法超越。
• wide & deep模型架构在所有损失函数下均优于逻辑回归，尤其在结合FN加权或FN校准损失时表现更优。