[Paper Review] A Network-based End-to-End Trainable Task-oriented Dialogue System

Tsung-Hsien Wen, David Vandyke+6 more|arXiv (Cornell University)|Apr 15, 2016

Topic Modeling32 references170 citations

TL;DR

The paper presents a neural end-to-end task-oriented dialogue system with an explicit belief tracker and database interface, plus a pipe-lined Wizard-of-Oz data collection method, achieving task completion in a restaurant domain with competitive metrics.

ABSTRACT

Teaching machines to accomplish tasks by conversing naturally with humans is challenging. Currently, developing task-oriented dialogue systems requires creating multiple components and typically this involves either a large amount of handcrafting, or acquiring costly labelled datasets to solve a statistical learning problem for each component. In this work we introduce a neural network-based text-in, text-out end-to-end trainable goal-oriented dialogue system along with a new way of collecting dialogue data based on a novel pipe-lined Wizard-of-Oz framework. This approach allows us to develop dialogue systems easily and without making too many assumptions about the task at hand. The results show that the model can converse with human subjects naturally whilst helping them to accomplish tasks in a restaurant search domain.

Motivation & Objective

• Motivate the need for end-to-end neural task-oriented dialogue models that can handle domain-specific tasks without heavy handcrafting.
• Propose a modular yet end-to-end trainable architecture combining intent encoding, belief tracking, a database operator, a policy network, and a generation network.
• Introduce a novel pipe-lined Wizard-of-Oz data collection framework to cheaply and quickly gather in-domain dialogue data.
• Demonstrate that the proposed model can achieve competitive task success and naturalness in a restaurant search domain with a modest dataset.

Proposed method

• Treat dialogue as a sequence-to-sequence problem augmented with dialogue history and DB search outcomes.
• Use an intent network (LSTM or CNN encoder) to produce a distributed representation of user input.
• Employ slot-value belief trackers with CNN-based feature extractors to maintain probabilistic slot values.
• Incorporate a database operator that queries the DB from belief tracker outputs to form a DB truth vector.
• Introduce a policy network that fuses intent, belief state, and DB match information to produce an action vector for response generation.
• Use a generation network to produce skeletal responses conditioned on the action vector and then lexicalise with DB values and delexicalised tokens.
• Optionally employ an attention-based extension to dynamically weight tracker beliefs during generation.

Experimental results

Research questions

• RQ1 Can an end-to-end trainable neural architecture with explicit belief tracking and a database interface achieve competitive task-oriented dialogue performance?
• RQ2 Does delexicalisation and weight tying reduce data requirements while preserving task success in a neural end-to-end model?
• RQ3 How does a crowdsourced pipe-lined Wizard-of-Oz data collection framework affect quality and cost of in-domain dialogue data?
• RQ4 What decoding strategies (average likelihood, weighted decoding, attention) best improve task success and naturalness?
• RQ5 How does the proposed model compare to a handcrafted modular baseline in terms of task success and user experience?

Key findings

• The end-to-end model with explicit belief trackers and a DB operator achieves meaningful task completion and natural interactions in a restaurant domain.
• CNN-based belief trackers generalize better to long-distance dependencies than N-gram trackers, yielding higher precision and recall.
• Attention-based generation and weighted decoding substantially improve task success, with attention providing the largest gain (~5 percentage points).
• The corpus-based evaluation shows the full model with attention and weighting achieves higher task success and BLEU scores than baselines.
• Human evaluation reports high subjective success (≈98%), with strong comprehension and naturalness ratings (>4/5) for the full model.
• Compared to a handcrafted modular system, the neural approach delivers higher user engagement and comparable task success.