[论文解读] Learning to Respond with Your Favorite Stickers: A Framework of Unifying Multi-Modality and User Preference in Multi-Turn Dialog

Shen Gao, Xiuying Chen+3 more|arXiv (Cornell University)|Nov 5, 2020

Multimodal Machine Learning Applications参考文献 94被引用 9

一句话总结

本文提出 PESRS（偏好增强的贴纸响应选择器），通过联合建模多轮对话上下文与用户贴纸偏好，实现更优的贴纸推荐。通过整合深度交互网络、基于键值记忆的用户偏好建模以及门控融合机制，PESRS 在大规模真实世界数据集上实现了最先进性能，在所有标准指标上均优于先前方法。

ABSTRACT

Stickers with vivid and engaging expressions are becoming increasingly popular in online messaging apps, and some works are dedicated to automatically select sticker response by matching the stickers image with previous utterances. However, existing methods usually focus on measuring the matching degree between the dialog context and sticker image, which ignores the user preference of using stickers. Hence, in this paper, we propose to recommend an appropriate sticker to user based on multi-turn dialog context and sticker using history of user. Two main challenges are confronted in this task. One is to model the sticker preference of user based on the previous sticker selection history. Another challenge is to jointly fuse the user preference and the matching between dialog context and candidate sticker into final prediction making. To tackle these challenges, we propose a \emph{Preference Enhanced Sticker Response Selector} (PESRS) model. Specifically, PESRS first employs a convolutional based sticker image encoder and a self-attention based multi-turn dialog encoder to obtain the representation of stickers and utterances. Next, deep interaction network is proposed to conduct deep matching between the sticker and each utterance. Then, we model the user preference by using the recently selected stickers as input, and use a key-value memory network to store the preference representation. PESRS then learns the short-term and long-term dependency between all interaction results by a fusion network, and dynamically fuse the user preference representation into the final sticker selection prediction. Extensive experiments conducted on a large-scale real-world dialog dataset show that our model achieves the state-of-the-art performance for all commonly-used metrics. Experiments also verify the effectiveness of each component of PESRS.

研究动机与目标

• 解决现有贴纸响应选择方法忽略用户特定贴纸偏好的局限性。
• 联合建模用户偏好与上下文-贴纸匹配，以在多轮对话中实现更精准的贴纸推荐。
• 构建一个动态融合用户偏好表征与对话交互特征及贴纸内容的框架。
• 提升真实消息应用中贴纸推荐的个性化与相关性。

提出的方法

• 基于自注意力机制的多轮对话编码器，用于捕捉对话轮次间的上下文表征。
• 卷积神经网络（CNN）将贴纸图像编码为密集嵌入向量。
• 深度交互网络计算话语词语与贴纸表征单元之间的共注意力，以建模细粒度匹配。
• 位置感知历史编码网络处理用户近期的贴纸选择，以捕捉使用中的时间模式。
• 基于键值的记忆网络根据当前上下文存储并检索用户偏好表征。
• 门控融合网络在最终贴纸预测前，动态结合交互结果与用户偏好特征。

实验结果

研究问题

• RQ1 如何从多轮对话中用户的历史选择行为中有效建模其贴纸偏好？
• RQ2 与仅基于上下文的匹配相比，引入用户偏好在多轮对话贴纸响应选择性能上能提升多少？
• RQ3 用户交互历史中的短期与长期依赖关系如何影响贴纸推荐的准确性？
• RQ4 各组件（如记忆网络、门控融合）对整体模型性能的贡献如何？

主要发现

• PESRS 在大规模真实世界对话数据集上的所有标准评估指标中均达到最先进性能。
• 引入用户偏好建模显著提升了预测准确率，超越仅依赖上下文-贴纸匹配的性能。
• 键值记忆网络有效捕捉了长期用户偏好，而位置感知编码器则建模了短期使用模式。
• 消融实验确认，各组件（尤其是门控融合与记忆网络）对性能提升均有显著贡献。
• 贴纸分类辅助任务达到 65.74% 的准确率，表明模型能从贴纸图像中捕捉语义信息。
• 贴纸中表情符号标签的分布显示高度多样性，但也表明因标注不一致可能引入标签噪声。