[Paper Review] UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2

Yunyi Yang, Yunhao Li+1 more|arXiv (Cornell University)|Dec 7, 2020

Topic ModelingComputer Science40 references32 citations

TL;DR

UBAR fine-tunes GPT-2 on full dialog sessions (user, belief state, DB result, system act, response) to build a fully end-to-end TOD system and achieves state-of-the-art results on MultiWOZ across generation, policy optimization, and end-to-end modeling. It evaluates with generated context to mirror real-life usage and demonstrates strong transfer to new domains with limited data.

ABSTRACT

This paper presents our task-oriented dialog system UBAR which models task-oriented dialogs on a dialog session level. Specifically, UBAR is acquired by fine-tuning the large pre-trained unidirectional language model GPT-2 on the sequence of the entire dialog session which is composed of user utterance, belief state, database result, system act, and system response of every dialog turn. Additionally, UBAR is evaluated in a more realistic setting, where its dialog context has access to user utterances and all content it generated such as belief states, system acts, and system responses. Experimental results on the MultiWOZ datasets show that UBAR achieves state-of-the-art performances in multiple settings, improving the combined score of response generation, policy optimization, and end-to-end modeling by 4.7, 3.5, and 9.4 points respectively. Thorough analyses demonstrate that the session-level training sequence formulation and the generated dialog context are essential for UBAR to operate as a fully end-to-end task-oriented dialog system in real life. We also examine the transfer ability of UBAR to new domains with limited data and provide visualization and a case study to illustrate the advantages of UBAR in modeling on a dialog session level.

Motivation & Objective

• Motivate moving from turn-level to session-level modeling for task-oriented dialogs to better reflect real-world usage.
• Propose a GPT-2 based model (UBAR) trained on full dialog sessions including belief states and system acts.
• Evaluate end-to-end, generation, and policy optimization settings with generated content in context.
• Analyze transferability to new domains with limited data and provide insights via visualization and case studies.

Proposed method

• Fine-tune DistilGPT-2 on sequences that concatenate entire dialog sessions: U, B, D, A, R for each turn.
• Delexicalize responses and use domain-adaptive, decoupled spans for belief states and system acts to improve generalization.
• Represent belief states and system acts with domain-slot/value and domain-act/span tokens to ground generation.
• Train with a standard language modeling objective on session-level sequences (no extra supervised objectives).
• Evaluate under three settings: ground-truth belief/state context for response generation, ground-truth belief/state for policy optimization, and end-to-end modeling with generated content.

Experimental results

Research questions

• RQ1 Can session-level training with intermediate information (belief states, system acts) improve end-to-end TOD performance?
• RQ2 Does evaluating with generated dialog context (instead of ground truth) better reflect real-life deployment?
• RQ3 How well does the model transfer to unseen domains with limited data?
• RQ4 What is the impact of dialog context length and content (ground-truth vs generated) on end-to-end TOD performance?

Key findings

• UBAR achieves state-of-the-art results across response generation, policy optimization, and end-to-end modeling on MultiWOZ 2.0/2.1.
• In end-to-end modeling, UBAR significantly improves the combined score compared to baselines, using all-generated context.
• Session-level sequence training and using generated dialog context are essential for real-life end-to-end TOD performance.
• UBAR shows transfer ability to new domains with limited data, especially with few-shot fine-tuning, though data hunger remains evident.
• Ablation studies show belief states and system acts in context are more critical than user utterances/responses for learning effective policy and grounding.