[Paper Review] What Should I Ask? Using Conversationally Informative Rewards for Goal-Oriented Visual Dialog

Pushkar Shukla, Carlos Elmadjian+4 more|arXiv (Cornell University)|Jul 28, 2019

Multimodal Machine Learning Applications23 references4 citations

TL;DR

This paper proposes an end-to-end goal-oriented visual dialogue system that uses Regularized Information Gain (RIG) with reinforcement learning to generate strategic, contextually relevant questions for identifying an object in an image. By modeling human inquiry through the Rational Speech Act framework, the method improves question relevance and reduces redundancy, achieving a new state-of-the-art accuracy of 67.19% on the GuessWhat?! dataset.

ABSTRACT

The ability to engage in goal-oriented conversations has allowed humans to gain knowledge, reduce uncertainty, and perform tasks more efficiently. Artificial agents, however, are still far behind humans in having goal-driven conversations. In this work, we focus on the task of goal-oriented visual dialogue, aiming to automatically generate a series of questions about an image with a single objective. This task is challenging since these questions must not only be consistent with a strategy to achieve a goal, but also consider the contextual information in the image. We propose an end-to-end goal-oriented visual dialogue system, that combines reinforcement learning with regularized information gain. Unlike previous approaches that have been proposed for the task, our work is motivated by the Rational Speech Act framework, which models the process of human inquiry to reach a goal. We test the two versions of our model on the GuessWhat?! dataset, obtaining significant results that outperform the current state-of-the-art models in the task of generating questions to find an undisclosed object in an image.

Motivation & Objective

• To develop a goal-oriented visual dialogue system that generates strategic, context-aware questions to identify an object in an image.
• To model human-like inquiry behavior using the Rational Speech Act framework, emphasizing information gain and question cost.
• To improve dialogue efficiency by minimizing redundant or uninformative questions through a regularized reward function.
• To outperform existing baselines in accuracy and coherence on the GuessWhat?! benchmark.
• To demonstrate that combining information gain with cost regularization leads to more effective and strategic question generation.

Proposed method

• Uses deep reinforcement learning to train a question generator that maximizes expected regularized information gain (RIG) while minimizing question cost.
• Introduces a novel reward function that balances information gain and the cost of asking a question, inspired by the Rational Speech Act framework.
• Employs a loss function based on RIG with a skewness coefficient to penalize synonymic or redundant questions and guide the model toward optimal strategies.
• Trains an end-to-end architecture with a question generator (QGen), a guesser, and an oracle, using visual features and dialogue history as input.
• Applies policy gradients to optimize the question generation policy, with ablation studies comparing RIG with standard information gain and baseline rewards.
• Uses the GuessWhat?! dataset for training and evaluation, with test splits for both new images and new objects to assess generalization.

Experimental results

Research questions

• RQ1 How can artificial agents generate goal-oriented questions that effectively reduce uncertainty about an object in an image?
• RQ2 What strategies do humans use during goal-driven visual inquiry, and can these be modeled computationally?
• RQ3 Can regularized information gain with question cost modeling improve the coherence and efficiency of visual dialogue systems?
• RQ4 How does RIG-based training compare to standard information gain or baseline reward functions in reducing redundant questions?
• RQ5 To what extent can end-to-end RL with RIG achieve human-like strategic behavior in visual question generation?

Key findings

• The proposed model achieves a new state-of-the-art accuracy of 67.19% on the GuessWhat?! dataset, surpassing previous methods.
• Using RIG as a reward function yields a 10.57% absolute improvement over standard information gain in accuracy on new images.
• The addition of a skewness coefficient improves performance by up to 2.8% when RIG is used as a loss function.
• The model generates only 0.36 repetitive questions per dialogue on average, significantly lower than the 0.82 of the baseline model by Strub et al. (2017).
• Ablation studies show that information gain alone is insufficient for optimal learning, but RIG with cost regularization leads to substantial gains.
• The model demonstrates strategic behavior and mutual consistency in question sequences, avoiding redundant or uninformative queries.