[论文解读] Visual Entailment: A Novel Task for Fine-Grained Image Understanding

Ning Xie, Farley Lai+2 more|arXiv (Cornell University)|Jan 20, 2019

Multimodal Machine Learning Applications参考文献 73被引用 162

一句话总结

This paper introduces Visual Entailment (VE), a cross-modal task where an image premise is used to determine if a natural language hypothesis is entailed, neutral, or contradicted, and presents the SNLI-VE dataset plus the Explainable Visual Entailment (EVE) model.

ABSTRACT

Existing visual reasoning datasets such as Visual Question Answering (VQA), often suffer from biases conditioned on the question, image or answer distributions. The recently proposed CLEVR dataset addresses these limitations and requires fine-grained reasoning but the dataset is synthetic and consists of similar objects and sentence structures across the dataset. In this paper, we introduce a new inference task, Visual Entailment (VE) - consisting of image-sentence pairs whereby a premise is defined by an image, rather than a natural language sentence as in traditional Textual Entailment tasks. The goal of a trained VE model is to predict whether the image semantically entails the text. To realize this task, we build a dataset SNLI-VE based on the Stanford Natural Language Inference corpus and Flickr30k dataset. We evaluate various existing VQA baselines and build a model called Explainable Visual Entailment (EVE) system to address the VE task. EVE achieves up to 71% accuracy and outperforms several other state-of-the-art VQA based models. Finally, we demonstrate the explainability of EVE through cross-modal attention visualizations. The SNLI-VE dataset is publicly available at https://github.com/ necla-ml/SNLI-VE.

研究动机与目标

• Motivate a cross-modal reasoning task that mitigates biases found in VQA datasets.
• Introduce Visual Entailment (VE) where image premises determine hypothesis truthfulness.
• Create SNLI-VE, a real-world image and SNLI-based hypothesis dataset for VE.
• Develop an interpretable VE model (EVE) that uses attention to reveal cross-modal reasoning.

提出的方法

• Define VE as a tri-class (entailment, neutral, contradiction) task with image premises and text hypotheses.
• Construct SNLI-VE by pairing Flickr30k images with SNLI hypotheses, ensuring disjoint partitions and bias considerations.
• Propose EVE, a dual-branch model with self-attention on text and image regions, plus text-image attention for cross-modal fusion.
• Compare EVE to VQA baselines and image-captioning baselines, using GloVe embeddings and Adam optimization.
• Provide attention visualizations to demonstrate model interpretability.

实验结果

研究问题

• RQ1 Can real-world images paired with SNLI-style hypotheses be reliably classified into entailment, neutral, or contradiction?
• RQ2 Do cross-modal attention mechanisms improve VE accuracy over VQA-based baselines?
• RQ3 Does an explainable attention-based VE model match or exceed state-of-the-art VQA performance on SNLI-VE?
• RQ4 How do image features (full maps vs ROIs) affect VE performance and interpretability?

主要发现

模型名称	Val Acc Overall (%)	C	N	E	Test Acc Overall (%)	C	N	E
Hypothesis Only	66.68	67.54	66.90	65.60	66.71	67.60	67.71	64.83
Image Captioning	67.83	66.61	69.23	67.65	67.67	66.25	70.69	66.08
Relational Network	67.56	67.86	67.80	67.02	67.55	67.29	68.86	66.50
Attention Top-Down	70.53	70.23	68.66	72.71	70.30	69.72	69.33	71.86
Attention Bottom-Up	69.34	71.26	70.10	66.67	68.90	70.52	70.96	65.23
EVE-Image*	71.56	71.04	70.55	73.10	71.16	71.56	70.52	71.39
EVE-ROI*	70.81	68.55	68.78	75.10	70.47	67.69	74.25	74.25

• EVE-Image achieves up to 71.56% validation accuracy and 71.16% test accuracy, outperforming several baselines.
• EVE-ROI achieves 70.81% validation and 70.47% test accuracy, showing self-attention and cross-modal attention help.</br>Attention-based models outperform standard VQA baselines on SNLI-VE.
• Hypothesis-only baselines reach ~66-67% accuracy, indicating inherent bias in the data and the need for image-guided reasoning.
• Image captioning as a premise source gives only marginal gains over hypothesis-only baselines, suggesting captions may miss crucial details for VE.
• Traditional Relational Networks offer limited gains on SNLI-VE, highlighting the need for richer cross-modal interaction modeling.
• The EVE model provides interpretable attention visualizations linking image regions to hypotheses.