[Paper Review] Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning

Tiancheng Zhao, Maxine Eskénazi|arXiv (Cornell University)|Jun 8, 2016

Speech and dialogue systemsComputer Science26 references131 citations

TL;DR

The paper presents an end-to-end framework for task-oriented dialogue using a Deep Recurrent Q-Network that interfaces with a database, and introduces a hybrid RL approach combining reinforcement and supervised learning to accelerate training, evaluated on a 20Q game.

ABSTRACT

This paper presents an end-to-end framework for task-oriented dialog systems using a variant of Deep Recurrent Q-Networks (DRQN). The model is able to interface with a relational database and jointly learn policies for both language understanding and dialog strategy. Moreover, we propose a hybrid algorithm that combines the strength of reinforcement learning and supervised learning to achieve faster learning speed. We evaluated the proposed model on a 20 Question Game conversational game simulator. Results show that the proposed method outperforms the modular-based baseline and learns a distributed representation of the latent dialog state.

Motivation & Objective

• Motivate replacing modular NLU, DST, and dialog policy with a single end-to-end learnable module for task-oriented dialogue.
• Enable interaction with structured databases through symbolic query-like actions within a reinforcement learning framework.
• Jointly learn state tracking and dialog policy to achieve better end-to-end performance and data efficiency.
• Leverage labeled data when available to accelerate learning while still functioning with weak or no DST labels.

Proposed method

• Use a variant of Deep Recurrent Q-Networks (DRQN) with an LSTM-based state tracker to accumulate dialog history into a belief-like state.
• Represent database interactions via special actions that modify a query hypothesis and return database observations and rewards.
• Implement multiple policy networks: one for verbal actions and others for slot-filling actions, sharing a common LSTM-derived state representation.
• Introduce a hybrid RL approach that incorporates supervised signals when state-tracking labels are available to speed up learning.
• Apply reward shaping and constrained action masking to improve sample efficiency and policy guidance.
• Optionally generate synthetic experiences from known database dynamics to accelerate learning (Dyna-like idea).

Experimental results

Research questions

• RQ1 Can end-to-end RL jointly learn dialog state tracking and dialog policy to outperform modular baselines?
• RQ2 Does interfacing with a database via symbolic actions improve end-to-end learning and information access?
• RQ3 Can supervised signals be integrated to speed up learning without sacrificing end-to-end optimization?
• RQ4 Is the learned dialog state representation meaningful and capable of supporting planning across turns?

Key findings

• End-to-end models outperform the modular baseline in win rate and interaction efficiency on 20Q.
• Baseline: 68.5% win rate, 12.2 turns on average; RL: 85.6% win rate, 21.6 turns; Hybrid-RL: 90.5% win rate, 19.22 turns.
• RL and Hybrid-RL converge faster and achieve higher performance than RL alone, with Hybrid-RL balancing exploration and state-tracking signals.
• The RL approach develops a different, higher-precision state-tracking behavior, often predicting unknown to avoid incorrect definite labels.
• learned dialog states (LSTM outputs) correlate with the true latent dialog state and can be used to reconstruct aspects like the number of guesses.
• The study confirms that end-to-end belief-state-like representations emerge from joint optimization and can approximate the true dialog state.