[Paper Review] Interactive Reinforcement Learning for Object Grounding via Self-Talking

Yan Zhu, Shaoting Zhang+1 more|arXiv (Cornell University)|Dec 2, 2017

Multimodal Machine Learning Applications9 references10 citations

TL;DR

This paper proposes an interactive reinforcement learning (IRL) framework for visual object grounding in dialog, where the question generator, answer model, and guesser are jointly trained with a shared reward function. The method significantly improves task success rate—reaching 82.9% on the GuessWhat?! benchmark—while introducing reward engineering to mitigate language drifting and enhance interpretability of generated dialogues.

ABSTRACT

Humans are able to identify a referred visual object in a complex scene via a few rounds of natural language communications. Success communication requires both parties to engage and learn to adapt for each other. In this paper, we introduce an interactive training method to improve the natural language conversation system for a visual grounding task. During interactive training, both agents are reinforced by the guidance from a common reward function. The parametrized reward function also cooperatively updates itself via interactions, and contribute to accomplishing the task. We evaluate the method on GuessWhat?! visual grounding task, and significantly improve the task success rate. However, we observe language drifting problem during training and propose to use reward engineering to improve the interpretability for the generated conversations. Our result also indicates evaluating goal-ended visual conversation tasks require semantic relevant metrics beyond task success rate.

Motivation & Objective

• To develop a more interactive and collaborative training framework for visual grounding in dialog, where all agents (question generator, answer model, guesser) are jointly optimized.
• To address the limitation of prior methods that only fine-tune the question generator while keeping other models fixed.
• To mitigate the language drifting problem—where generated dialogues become uninterpretable—during interactive reinforcement learning.
• To improve the interpretability of generated dialogues without sacrificing task success rate, using reward engineering.
• To demonstrate that task success rate alone is insufficient for evaluating goal-ended visual conversation systems, and that semantic quality metrics are essential.

Proposed method

• Jointly train the question generator, answer model, and guesser using a common reward function via reinforcement learning in a dynamic environment.
• Use a parameterized guesser model as the reward function, which dynamically updates during training to adapt to the evolving policies of the other agents.
• Implement a seq2seq model with global dot-product attention for the question generator to improve long-range reasoning in dialog.
• Apply reward engineering via heuristic pruning: remove questions with repetitive phrases or near-duplicate sequences to enforce natural language patterns.
• Use multinomial sampling during inference and fine-tune models with Adam optimizer (initial learning rate 1e-4, batch size 64).
• Pre-train models supervisedly before fine-tuning with reinforcement learning to stabilize training and achieve comparable baseline performance.

Experimental results

Research questions

• RQ1 Can interactive reinforcement learning that jointly trains all three agents (question generator, answer model, guesser) significantly improve task success rate in visual grounding compared to prior methods?
• RQ2 To what extent does language drifting emerge during interactive training, and how does it affect the interpretability of generated dialogues?
• RQ3 Can reward engineering techniques that prune unnatural or redundant dialog turns improve the readability and semantic quality of generated conversations?
• RQ4 Is task success rate alone sufficient to evaluate goal-ended visual conversation systems, or are additional semantic metrics required?
• RQ5 How does joint training of the answer model and guesser affect the quality and consistency of generated answers compared to only training the question generator?

Key findings

• The proposed IRL QAG model achieves a task success rate of 82.9% on the GuessWhat?! benchmark, significantly outperforming the previous best result of 60.3% and approaching human-level performance (84.4%).
• The IRL QAG model exhibits severe language drifting, with 59.0% of human-annotated answers judged as disagreeing with human judgment, indicating a semantic gap between AI-generated and natural language.
• Reward engineering via pruning (IRL-prune QAG) improves question interpretability, raising the average question quality ranking from 4.98 to 3.53, though at a slight cost to task success rate (81.3%).
• The answer model in IRL QAG generates more 'n/a' responses even when 'yes' or 'no' would be more appropriate, indicating degradation in answer quality due to language drift.
• Jointly training the answer model and guesser (IRL AG) improves answer quality compared to only training the question generator, suggesting better alignment between question and answer policies.
• Human evaluation confirms that models trained with interactive RL generate dialogues that are less interpretable than natural language, highlighting the need for semantic evaluation beyond task success rate.