[Paper Review] CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog

Satwik Kottur, José M. F. Moura+3 more|arXiv (Cornell University)|Mar 7, 2019

Multimodal Machine Learning Applications19 references50 citations

TL;DR

This paper introduces CLEVR-Dialog, a large synthetic diagnostic dataset for studying multi-round reasoning in visually grounded dialog, with fully annotated scene graphs, and benchmarks coreference resolution in visual dialog.

ABSTRACT

Visual Dialog is a multimodal task of answering a sequence of questions grounded in an image, using the conversation history as context. It entails challenges in vision, language, reasoning, and grounding. However, studying these subtasks in isolation on large, real datasets is infeasible as it requires prohibitively-expensive complete annotation of the 'state' of all images and dialogs. We develop CLEVR-Dialog, a large diagnostic dataset for studying multi-round reasoning in visual dialog. Specifically, we construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset. This combination results in a dataset where all aspects of the visual dialog are fully annotated. In total, CLEVR-Dialog contains 5 instances of 10-round dialogs for about 85k CLEVR images, totaling to 4.25M question-answer pairs. We use CLEVR-Dialog to benchmark performance of standard visual dialog models; in particular, on visual coreference resolution (as a function of the coreference distance). This is the first analysis of its kind for visual dialog models that was not possible without this dataset. We hope the findings from CLEVR-Dialog will help inform the development of future models for visual dialog. Our dataset and code are publicly available.

Motivation & Objective

• Motivate studying vision-language-Reasoning subproblems (vision, language, grounding, memory) in visual dialog in isolation.
• Provide a large, exhaustively annotated diagnostic dataset by grounding a dialog grammar in CLEVR scene graphs.
• Enable analysis of how dialog history and coreference distance affect model performance.
• Benchmark baseline visual dialog models and diagnose visual coreference resolution capabilities.

Proposed method

• Construct a dialog grammar grounded in CLEVR scene graphs to generate fully annotated visual dialogs.
• Use primitives (Sample, Unique, Count, Group, Filter, Exist, Relate) to build caption and question templates.
• Generate captions and questions with a rule-based grammar and beam search to ensure diverse, non-redundant dialogs.
• Annotate each CLEVR image with a complete scene graph and track history-dependent dialog states.
• Benchmark neural models (including CorefNMN) on CLEVR-Dialog to analyze performance as a function of coreference distance and history dependency.

Experimental results

Research questions

• RQ1 How does visual coreference resolution performance degrade as coreference distance increases in visual dialog?
• RQ2 How do different visual dialog models leverage dialog history to answer questions grounded in images?
• RQ3 What is the impact of history-dependent questions on overall accuracy across models?
• RQ4 Which model architectures best handle coreference and history-based reasoning in visual dialog?

Key findings

• CLEVR-Dialog enables large-scale analysis of visual coreference and history dependence not feasible with real-world datasets.
• CorefNMN achieves 68.0% accuracy on CLEVR-Dialog, outperforming several baselines and demonstrating the value of explicit coreference modeling.
• History-dependent questions (especially those requiring long-range coreference) reveal that some models (e.g., HRE-QIH) can outperform coreference-focused architectures on all questions by leveraging dialog-level summarization.
• CLEVR-Dialog shows mean coreference distance of 3.2 rounds and a broader distance range (1 to 10), indicating more challenging multi-round reasoning than MNIST-Dialog.
• Dataset statistics show 85k CLEVR images with 3.5M train QA pairs and 0.75M val QA pairs, across 5 instance dialogs of 10 rounds per image.
• Various question types (count, exist, seek) and a rich history dependency distribution demonstrate higher linguistic diversity and reasoning demands than prior synthetic datasets.