[Paper Review] Learning through Dialogue Interactions by Asking Questions

Jiwei Li, Alexander Miller+3 more|arXiv (Cornell University)|Dec 15, 2016

Speech and dialogue systemsComputer Science72 citations

TL;DR

This paper proposes a framework for training dialogue agents to improve question-answering performance by learning to ask clarifying questions during interaction. Using a simulated movie QA environment and real human data via Mechanical Turk, the authors demonstrate that agents trained to ask questions—both offline and online—achieve significantly higher accuracy than those relying only on fixed responses, especially in knowledge-scarce or ambiguous scenarios.

ABSTRACT

A good dialogue agent should have the ability to interact with users by both responding to questions and by asking questions, and importantly to learn from both types of interaction. In this work, we explore this direction by designing a simulator and a set of synthetic tasks in the movie domain that allow such interactions between a learner and a teacher. We investigate how a learner can benefit from asking questions in both offline and online reinforcement learning settings, and demonstrate that the learner improves when asking questions. Finally, real experiments with Mechanical Turk validate the approach. Our work represents a first step in developing such end-to-end learned interactive dialogue agents.

Motivation & Objective

• To investigate how dialogue agents can learn from interactive feedback by asking questions, rather than relying solely on fixed training responses.
• To address three key failure modes in dialogue: misunderstanding surface forms, struggling with reasoning, and lacking necessary knowledge.
• To design a simulator and synthetic tasks in the movie QA domain to enable structured study of question-asking as a learning mechanism.
• To evaluate the impact of question-asking in both offline supervised and online reinforcement learning settings.
• To validate the approach using real human teachers on Amazon Mechanical Turk, confirming results generalize beyond simulation.

Proposed method

• Designed a dialogue simulator with three synthetic tasks: Question Clarification, Knowledge Operation, and Knowledge Acquisition, each targeting a distinct failure mode in dialogue understanding.
• Used the WikiMovies dataset as a knowledge base and adapted it to create scenarios where agents must ask questions to resolve ambiguity, retrieve relevant facts, or acquire missing knowledge.
• Implemented a memory network-based model (MemN2N) with context-aware attention (Cont-MemN2N) to better handle unseen words and complex reasoning by leveraging dialogue history.
• Evaluated offline learning by training agents to ask questions during inference (TestAQ) and compared to models that never ask (TestQA), using both synthetic and real human data.
• Applied online reinforcement learning with a cost function for question-asking, enabling agents to learn optimal timing for asking questions based on expected reward and cost trade-offs.
• Conducted human evaluation via Mechanical Turk, collecting real teacher responses to validate the model’s performance on real-world dialogue complexity and variability.

Experimental results

Research questions

• RQ1 Can a dialogue agent improve its question-answering performance by learning to ask questions during interaction, rather than relying only on fixed responses?
• RQ2 How does question-asking impact performance in scenarios involving surface form ambiguity, reasoning complexity, and knowledge gaps?
• RQ3 What is the optimal strategy for when to ask questions in online reinforcement learning, given a cost for each question?
• RQ4 Does the benefit of question-asking generalize from simulated data to real human-teacher interactions?
• RQ5 How does context-aware modeling (Cont-MemN2N) enhance the agent’s ability to ask relevant questions and improve performance?

Key findings

• In the Missing Answer Entity and Missing Everything tasks, agents that did not ask questions at test time (TestQA) achieved accuracy below 0.01, demonstrating the critical need for question-asking in knowledge-scarce scenarios.
• Agents using learned question-asking strategies (TestModelAQ) achieved performance comparable to those asking correct questions (TestAQ) and significantly better than those not asking at all (TestQA), even when asking potentially irrelevant questions.
• Cont-MemN2N outperformed vanilla-MemN2N by leveraging dialogue context to better distinguish correct answers, especially when encountering unfamiliar words.
• In online reinforcement learning, poor students asked questions more frequently than good students, especially at low cost, and benefited more from asking, leading to higher accuracy gains under high cost penalties.
• On Mechanical Turk, the TrainAQ+TestAQ setting (asking questions at both training and inference) achieved the highest performance across all tasks, confirming that interaction through questioning enhances learning in real human interactions.
• Despite lower absolute performance due to noise and smaller training data, the same trends held on real human data, validating the robustness and generalizability of the approach.