[Paper Review] iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability

Aman Chadha, Vinija Jain|arXiv (Cornell University)|Jun 25, 2021

Multimodal Machine Learning Applications54 references4 citations

TL;DR

iReason is a multimodal framework that enhances commonsense reasoning in AI by jointly leveraging videos and natural language captions to infer causal relationships, using a canonical frame detection module for video input and a causal rationalization module for interpretability. It outperforms state-of-the-art models like VCC and BERT/GPT-2 across multiple benchmarks, demonstrating superior performance in causal event prediction and interpretability.

ABSTRACT

Causality knowledge is vital to building robust AI systems. Deep learning models often perform poorly on tasks that require causal reasoning, which is often derived using some form of commonsense knowledge not immediately available in the input but implicitly inferred by humans. Prior work has unraveled spurious observational biases that models fall prey to in the absence of causality. While language representation models preserve contextual knowledge within learned embeddings, they do not factor in causal relationships during training. By blending causal relationships with the input features to an existing model that performs visual cognition tasks (such as scene understanding, video captioning, video question-answering, etc.), better performance can be achieved owing to the insight causal relationships bring about. Recently, several models have been proposed that have tackled the task of mining causal data from either the visual or textual modality. However, there does not exist widespread research that mines causal relationships by juxtaposing the visual and language modalities. While images offer a rich and easy-to-process resource for us to mine causality knowledge from, videos are denser and consist of naturally time-ordered events. Also, textual information offers details that could be implicit in videos. We propose iReason, a framework that infers visual-semantic commonsense knowledge using both videos and natural language captions. Furthermore, iReason's architecture integrates a causal rationalization module to aid the process of interpretability, error analysis and bias detection. We demonstrate the effectiveness of iReason using a two-pronged comparative analysis with language representation learning models (BERT, GPT-2) as well as current state-of-the-art multimodal causality models.

Motivation & Objective

• Address the gap in multimodal causal reasoning by integrating both visual and linguistic modalities to infer commonsense knowledge.
• Improve model robustness and interpretability by incorporating causal relationships into vision-and-language tasks.
• Overcome limitations of existing models that rely on static images, random frame pairs, or lack interpretability.
• Enable error analysis and bias detection through a dedicated causal rationalization module.
• Demonstrate the universal applicability of causal knowledge infusion across diverse downstream vision-and-language tasks.

Proposed method

• Utilizes videos as native input with a canonical frame detection module to identify temporally salient events for causal reasoning.
• Integrates natural language captions to enrich visual event representations and provide linguistic context for causal inference.
• Employs a causal rationalization module that generates human-readable justifications for predicted causal relationships.
• Combines visual features from a pre-trained object detector with contextual embeddings from BERT or GPT-2 to encode multimodal inputs.
• Trains a joint multimodal encoder to predict causal relationships between event pairs, using contrastive learning to distinguish causal from non-causal pairs.
• Applies ablation studies to isolate the contribution of visual, linguistic, and multimodal contexts in causal prediction.

Experimental results

Research questions

• RQ1 Can joint modeling of videos and natural language captions improve the accuracy of causal reasoning compared to unimodal or non-causal baselines?
• RQ2 How does the inclusion of a causal rationalization module enhance interpretability and support error analysis in multimodal models?
• RQ3 To what extent do visual and linguistic modalities complement each other in inferring commonsense causal relationships?
• RQ4 Does canonical frame detection from video sequences lead to better causal inference than random frame selection or image-based models?
• RQ5 Can the integration of causal knowledge improve performance across diverse downstream tasks like video question-answering and dense captioning?

Key findings

• iReason outperforms VCC zhang2020learning on all metrics, achieving 38.71% R@5 and 65.12% R@10 on the VQA-CAUSAL dataset, indicating superior causal event detection.
• iReason achieves 9.21% R@1 on the BERT-based setup and 8.90% R@1 on the GPT-2-based setup, significantly outperforming random guess (2.13%) and standard language models.
• The model with both visual and linguistic context achieves the highest performance, showing that multimodal context is more effective than unimodal inputs.
• Ablation studies confirm that using only linguistic context outperforms using only visual context, but the combination yields the best results.
• The causal rationalization module successfully generates interpretable justifications for predictions, enabling error analysis and bias detection.
• iReason demonstrates state-of-the-art performance across diverse tasks, including dense video captioning, video question-answering, and scene understanding, with consistent gains over prior models.