[论文解读] iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability

Aman Chadha, Vinija Jain|arXiv (Cornell University)|Jun 25, 2021

Multimodal Machine Learning Applications参考文献 54被引用 4

一句话总结

iReason 是一种多模态框架，通过联合利用视频和自然语言字幕来推断因果关系，从而增强人工智能中的常识推理能力。该框架采用规范帧检测模块处理视频输入，并通过因果推理解释模块提升可解释性。在多个基准测试中，iReason 的表现优于 SOTA 模型（如 VCC 和 BERT/GPT-2），在因果事件预测和可解释性方面展现出卓越性能。

ABSTRACT

Causality knowledge is vital to building robust AI systems. Deep learning models often perform poorly on tasks that require causal reasoning, which is often derived using some form of commonsense knowledge not immediately available in the input but implicitly inferred by humans. Prior work has unraveled spurious observational biases that models fall prey to in the absence of causality. While language representation models preserve contextual knowledge within learned embeddings, they do not factor in causal relationships during training. By blending causal relationships with the input features to an existing model that performs visual cognition tasks (such as scene understanding, video captioning, video question-answering, etc.), better performance can be achieved owing to the insight causal relationships bring about. Recently, several models have been proposed that have tackled the task of mining causal data from either the visual or textual modality. However, there does not exist widespread research that mines causal relationships by juxtaposing the visual and language modalities. While images offer a rich and easy-to-process resource for us to mine causality knowledge from, videos are denser and consist of naturally time-ordered events. Also, textual information offers details that could be implicit in videos. We propose iReason, a framework that infers visual-semantic commonsense knowledge using both videos and natural language captions. Furthermore, iReason's architecture integrates a causal rationalization module to aid the process of interpretability, error analysis and bias detection. We demonstrate the effectiveness of iReason using a two-pronged comparative analysis with language representation learning models (BERT, GPT-2) as well as current state-of-the-art multimodal causality models.

研究动机与目标

• 通过整合视觉与语言模态，弥合多模态因果推理中的研究空白，以推断常识知识。
• 通过将因果关系融入视觉-语言任务，提升模型的鲁棒性与可解释性。
• 克服现有模型依赖静态图像、随机帧对或缺乏可解释性的局限。
• 通过专用的因果推理解释模块，实现错误分析与偏差检测。
• 证明因果知识注入在多样化下游视觉-语言任务中的通用适用性。

提出的方法

• 以视频作为原生输入，利用规范帧检测模块识别时序上显著的事件，用于因果推理。
• 整合自然语言字幕，丰富视觉事件表征，并为因果推理提供语言上下文。
• 采用因果推理解释模块，生成人类可读的预测理由。
• 结合预训练目标检测器的视觉特征与 BERT 或 GPT-2 的上下文嵌入，编码多模态输入。
• 训练联合多模态编码器，预测事件对之间的因果关系，使用对比学习区分因果与非因果对。
• 通过消融研究，分离视觉、语言和多模态上下文在因果预测中的贡献。

实验结果

研究问题

• RQ1 与单模态或非因果基线相比，联合建模视频与自然语言字幕是否能提升因果推理的准确性？
• RQ2 因果推理解释模块的引入在多模态模型中如何增强可解释性，并支持错误分析？
• RQ3 视觉与语言模态在推断常识性因果关系方面，其互补性在多大程度上体现？
• RQ4 与随机帧选择或基于图像的模型相比，从视频序列中进行规范帧检测是否能带来更优的因果推理？
• RQ5 因果知识的整合是否能提升多样化下游任务（如视频问答与密集字幕生成）的性能？

主要发现

• iReason 在所有指标上均优于 VCC zhang2020learning，在 VQA-CAUSAL 数据集上达到 38.71% 的 R@5 和 65.12% 的 R@10，表明其在因果事件检测方面表现更优。
• 在基于 BERT 的设置中，iReason 达到 9.21% 的 R@1，在基于 GPT-2 的设置中达到 8.90% 的 R@1，显著优于随机猜测（2.13%）和标准语言模型。
• 同时使用视觉与语言上下文的模型性能最高，表明多模态上下文比单模态输入更有效。
• 消融研究证实，仅使用语言上下文的表现优于仅使用视觉上下文，但两者结合可获得最佳结果。
• 因果推理解释模块成功生成可解释的预测理由，支持错误分析与偏差检测。
• iReason 在多样化任务中均达到 SOTA 性能，包括密集视频字幕生成、视频问答和场景理解，且在各项任务中持续优于先前模型。