[论文解读] M2P2: Multimodal Persuasion Prediction using Adaptive Fusion

Chongyang Bai, Haipeng Chen+3 more|arXiv (Cornell University)|Jun 3, 2020

Misinformation and Its Impacts被引用 5

一句话总结

M2P2 提出了一种新颖的自适应融合框架，用于多模态说服力预测，利用语音、视觉和语言模态解决辩论结果预测（DOP）和新提出的说服力强度预测（IPP）问题。在 DOP 上，其性能优于先前基线模型 2%–3.7%，在 IPP 上 MSE 指标提升超过 25%，采用双模块架构，通过单模态参考模型学习模态对齐与异质性。

ABSTRACT

Identifying persuasive speakers in an adversarial environment is a critical task. In a national election, politicians would like to have persuasive speakers campaign on their behalf. When a company faces adverse publicity, they would like to engage persuasive advocates for their position in the presence of adversaries who are critical of them. Debates represent a common platform for these forms of adversarial persuasion. This paper solves two problems: the Debate Outcome Prediction (DOP) problem predicts who wins a debate while the Intensity of Persuasion Prediction (IPP) problem predicts the change in the number of votes before and after a speaker speaks. Though DOP has been previously studied, we are the first to study IPP. Past studies on DOP fail to leverage two important aspects of multimodal data: 1) multiple modalities are often semantically aligned, and 2) different modalities may provide diverse information for prediction. Our M2P2 (Multimodal Persuasion Prediction) framework is the first to use multimodal (acoustic, visual, language) data to solve the IPP problem. To leverage the alignment of different modalities while maintaining the diversity of the cues they provide, M2P2 devises a novel adaptive fusion learning framework which fuses embeddings obtained from two modules -- an alignment module that extracts shared information between modalities and a heterogeneity module that learns the weights of different modalities with guidance from three separately trained unimodal reference models. We test M2P2 on the popular IQ2US dataset designed for DOP. We also introduce a new dataset called QPS (from Qipashuo, a popular Chinese debate TV show ) for IPP. M2P2 significantly outperforms 4 recent baselines on both datasets.

研究动机与目标

• 为解决现有方法在说服力强度预测（IPP）这一新问题上的缺失，该问题通过票数变化衡量个体发言人的说服力。
• 通过更有效地利用多模态数据（语音、视觉、语言）提升辩论结果预测（DOP）的性能，超越以往方法。
• 开发一种框架，同时捕捉模态间的对齐（共享）信息与模态特异（异质）线索。
• 创建并发布源自中国辩论节目《奇葩说》的全新数据集 QPS，以支持未来在说服力预测领域的研究。

提出的方法

• M2P2 采用双模块架构：对齐模块用于提取跨模态的共享表征，异质性模块用于学习模态特异性权重。
• 异质性模块通过三个独立训练的单模态参考模型（语音、视觉、语言）引导，实现在融合过程中动态分配各模态的重要性。
• 通过结合对齐模块与异质性模块的嵌入表示，实现自适应融合，使模型能够聚焦于可靠且信息丰富的模态。
• 对于 IQ2US，使用通用句子编码器生成句子嵌入；对于 QPS，通过微调的 LSTMs 学习 128 维句子嵌入。
• 该框架在多模态序列上端到端训练，损失函数针对 DOP（分类任务）和 IPP（回归任务）进行定制。
• 消融研究与模态权重分析验证了模型动态优先选择信息丰富模态的能力，尤其在语言和语音模态上表现突出。

Figure 1 : In multimodal content, the modalities are semantically aligned. This example shows a case where the visual modality (facial expressions) and the language modality (the content of the speech) are closely aligned.

实验结果

研究问题

• RQ1 多模态深度学习框架能否在团队层面结果之外，有效预测个体发言人说服力（以票数变化衡量，即 IPP）？
• RQ2 当多模态（语音、视觉、语言）同时表现出语义对齐与信息异质性时，模型如何实现自适应融合？
• RQ3 在低资源场景下，通过异质性模块学习模态特异性重要性是否优于注意力机制？
• RQ4 单模态参考模型在说服力预测的多模态融合框架中，能在多大程度上提升性能？
• RQ5 源自真实辩论电视节目的新数据集 QPS，能否实现对 IPP 模型的可靠且可泛化的评估？

主要发现

• 在 IQ2US 数据集上，M2P2 相较近期基线模型将 DOP 准确率提升 2%–3.7%，且具有统计显著性（p < 0.05），表明其性能更优。
• 在新发布的 QPS 数据集上，M2P2 相较适配的基线模型将 IPP 的均方误差（MSE）降低超过 25%，且 p < 0.01，表明其泛化能力极强。
• 语言模态对预测贡献最大，其次为语音模态，而视觉模态效果最弱，该结论在消融实验与模态权重分析中得到验证。
• 在 IQ2US 上，使用词级嵌入相比句级嵌入性能提升 1.4%；在 QPS 上 MSE 降低 20%，表明细粒度语义结构具有重要意义。
• 异质性模块因参数更少，在小样本数据集上泛化能力更强，避免过拟合，因此优于注意力机制。
• 该框架实现了准确的实时说服力预测，如图 LABEL:fig:real_time_prediction 所示，M2P2 的输出能紧密跟踪辩论过程中真实票数变化。

(a) There are cases where the visual modality is noisy, while the language modality is clean. In 4 consecutive frames when the woman is speaking, the face of a man appears (see frames 2 and 3) and the man’s face is incorrectly assumed to be the woman’s. The language modality, however, is correct.