[论文解读] What We Know About Using Non-Engagement Signals in Content Ranking

Tom Cunningham, Sana Pandey+8 more|arXiv (Cornell University)|Feb 9, 2024

Sentiment Analysis and Opinion Mining被引用 7

一句话总结

本文回顾了一场为期一天的专家研讨会关于内容排序中的非参与信号的证据，结果显示基于参与度的排名在留存方面有效，但可以通过质量代理、逐项调查和多样化的参与信号来提升，而用户控制的使用存在混合，且对态度的影响有限。

ABSTRACT

Many online platforms predominantly rank items by predicted user engagement. We believe that there is much unrealized potential in including non-engagement signals, which can improve outcomes both for platforms and for society as a whole. Based on a daylong workshop with experts from industry and academia, we formulate a series of propositions and document each as best we can from public evidence, including quantitative results where possible. There is strong evidence that ranking by predicted engagement is effective in increasing user retention. However retention can be further increased by incorporating other signals, including item "quality" proxies and asking users what they want to see with "item-level" surveys. There is also evidence that "diverse engagement" is an effective quality signal. Ranking changes can alter the prevalence of self-reported experiences of various kinds (e.g. harassment) but seldom have large enough effects on attitude measures like user satisfaction, well-being, polarization etc. to be measured in typical experiments. User controls over ranking often have low usage rates, but when used they do correlate well with quality and item-level surveys. There was no strong evidence on the impact of transparency/explainability on retention. There is reason to believe that generative AI could be used to create better quality signals and enable new kinds of user controls.

研究动机与目标

• 促使对内容 ranking 中超出预测参与度的非参与信号进行探索。
• 总结公开证据与定量结果，阐明非参与信号如何影响留存、质量感知和用户体验。
• 识别在整合非参与信号时潜在的控制机制、透明性及未来方向。

提出的方法

• 综合来自为期一天的产业-学术界研讨会的发现。
• 记录建议并在可获得的情况下用公开证据和定量结果予以支持。
• 讨论诸如项质量代理、逐项调查和多样化参与等信号。
• 考虑用户对排名的控制及其与感知质量和态度之间的关系。

实验结果

研究问题

• RQ1 哪些非参与信号有潜力提升超出预测参与度的内容排名？
• RQ2 非参与信号如何影响留存、质量感知和用户体验？
• RQ3 用户控制和透明度对参与、质量与态度的影响是什么？
• RQ4 生成式人工智能是否有助于在排名中创建更好的信号和用户控制？

主要发现

• 基于预测参与度的排名能有效提升留存。
• 如项质量代理和逐项调查等非参与信号可进一步提升留存。
• 多样化的参与是一个有效的质量信号。
• 排名变动可能影响自我报告的体验（如骚扰），但通常实验中很少改变如满意度或两极化等大型态度测量。
• 对排名的用户控制使用率低，但在使用时与质量和逐项调查相关。
• 关于透明度/可解释性影响留存的证据并不充分。
• 生成式人工智能可能有助于创建更好的信号并实现新的用户控制。