[Paper Review] FlipDial: A Generative Model for Two-Way Visual Dialogue

Daniela Massiceti, N. Siddharth+2 more|arXiv (Cornell University)|Feb 11, 2018

Multimodal Machine Learning Applications24 references16 citations

TL;DR

FlipDial proposes a generative, convolutional neural network-based model for two-way visual dialogue, enabling a single agent to alternately generate questions and answers grounded in an image and its caption. It achieves state-of-the-art performance on the one-way visual dialogue (1vd) task and establishes the first baseline for two-way visual dialogue (2vd), introducing novel metrics for evaluating question relevance and latent dialogue diversity.

ABSTRACT

We present FlipDial, a generative model for visual dialogue that simultaneously plays the role of both participants in a visually-grounded dialogue. Given context in the form of an image and an associated caption summarising the contents of the image, FlipDial learns both to answer questions and put forward questions, capable of generating entire sequences of dialogue (question-answer pairs) which are diverse and relevant to the image. To do this, FlipDial relies on a simple but surprisingly powerful idea: it uses convolutional neural networks (CNNs) to encode entire dialogues directly, implicitly capturing dialogue context, and conditional VAEs to learn the generative model. FlipDial outperforms the state-of-the-art model in the sequential answering task (one-way visual dialogue) on the VisDial dataset by 5 points in Mean Rank using the generated answers. We are the first to extend this paradigm to full two-way visual dialogue, where our model is capable of generating both questions and answers in sequence based on a visual input, for which we propose a set of novel evaluation measures and metrics.

Motivation & Objective

• To develop a unified generative model capable of both answering and posing questions in visual dialogue, moving beyond one-sided answer-only models.
• To enable realistic, autoregressive dialogue generation where predicted dialogue history is used at test time, simulating real-world deployment.
• To propose and validate novel evaluation metrics for two-way visual dialogue, addressing the lack of standardized benchmarks for such models.
• To explore the effectiveness of convolutional neural networks in modeling sequential dependencies across dialogue turns, implicitly capturing context.
• To demonstrate that conditional variational autoencoders (cVAEs) with CNN-based sequence encoding can generate diverse, contextually relevant, and semantically coherent dialogues.

Proposed method

• FlipDial uses a conditional variational autoencoder (cVAE) framework to learn a deep generative model of visual dialogue, conditioning on image, caption, and dialogue history.
• It employs convolutional neural networks (CNNs) to encode entire dialogue sequences as tensors, implicitly modeling long-range dependencies and context across turns.
• The model generates answers and questions autoregressively by sampling from a conditional prior over latent variables, with inference amortized over the training data.
• For two-way dialogue (2vd), the model alternates between generating questions and answers in a single forward pass, conditioned on the image, caption, and prior dialogue history.
• It introduces two novel evaluation metrics: $\text{sim}_{\bm{c},\bm{q}}$, measuring semantic similarity between generated questions and the image caption, and $\text{sim}_{\circlearrowleft}$, measuring KL divergence in latent space between generated and ground-truth dialogues.
• The model is trained and evaluated under a realistic setting where predicted (not ground-truth) dialogue history is used at test time, enhancing practical applicability.

Experimental results

Research questions

• RQ1 Can a single generative model effectively alternate between asking and answering questions in a visually grounded dialogue, maintaining coherence and relevance?
• RQ2 How effective is a CNN-based sequence encoding approach in modeling sequential dependencies across dialogue turns in visual dialogue?
• RQ3 To what extent does autoregressive modeling with increasing depth improve the diversity and quality of generated dialogues in two-way visual dialogue?
• RQ4 Can novel, automatic metrics like $\text{sim}_{\bm{c},\bm{q}}$ and $\text{sim}_{\circlearrowleft}$ reliably assess the quality of generated visual dialogues, especially in two-way settings?
• RQ5 Does using predicted dialogue history at test time—rather than ground-truth—adversely affect model performance, and can the model still generate high-quality dialogues under this constraint?

Key findings

• FlipDial achieves a Mean Rank of 14.8 on the VisDial 1vd benchmark using generated answers, outperforming the previous SOTA model by 5 points.
• The model sets a new state-of-the-art baseline for two-way visual dialogue (2vd), a task previously unaddressed in the literature with a unified generative framework.
• As the number of autoregressive layers increases from 8 to 10, the latent dialogue dispersion metric $\text{sim}_{\circlearrowleft}$ increases from 31.50 to 44.34, indicating improved diversity in generated dialogues.
• The question relevance metric $\text{sim}_{\bm{c},\bm{q}}$ shows consistent performance across models, with values around 0.41–0.49, indicating that generated questions are semantically aligned with the image caption.
• In iterative evaluation (using predicted history), the cross-entropy loss decreases from 31.18 (B model) to 24.93 (B_AR10), showing that deeper autoregressive modeling improves generation quality under realistic test conditions.
• Qualitative analysis confirms that FlipDial generates diverse, coherent, and contextually relevant question-answer pairs, with no obvious repetition or nonsensical outputs in most generations.