[Paper Review] Hybrid Generative-Retrieval Transformers for Dialogue Domain Adaptation

Igor Shalyminov, Alessandro Sordoni+2 more|arXiv (Cornell University)|Mar 3, 2020

Topic Modeling36 references4 citations

TL;DR

This paper proposes a hybrid generative-retrieval transformer model fine-tuned on GPT-2 for few-shot dialogue domain adaptation, combining sequence generation with retrieval-based response selection. It achieves state-of-the-art performance on the Multi-domain Wizard-of-Oz (MultiWOZ) dataset in human evaluation, outperforming the second-place system by over 4% in win rate.

ABSTRACT

Domain adaptation has recently become a key problem in dialogue systems research. Deep learning, while being the preferred technique for modeling such systems, works best given massive training data. However, in the real-world scenario, such resources aren't available for every new domain, so the ability to train with a few dialogue examples can be considered essential. Pre-training on large data sources and adapting to the target data has become the standard method for few-shot problems within the deep learning framework. In this paper, we present the winning entry at the fast domain adaptation task of DSTC8, a hybrid generative-retrieval model based on GPT-2 fine-tuned to the multi-domain MetaLWOz dataset. Robust and diverse in response generation, our model uses retrieval logic as a fallback, being SoTA on MetaLWOz in human evaluation (>4% improvement over the 2nd place system) and attaining competitive generalization performance in adaptation to the unseen MultiWOZ dataset.

Motivation & Objective

• Address the challenge of data efficiency in training goal-oriented dialogue systems for new domains with limited in-domain data.
• Overcome the limitations of purely generative or retrieval-based models in few-shot adaptation scenarios.
• Develop a robust, diverse response generation system that generalizes well to unseen domains.
• Leverage transfer learning from large-scale pre-trained language models to improve performance on low-resource domains.
• Achieve superior human-rated response quality and competitive automatic metric scores in few-shot domain adaptation.

Proposed method

• Fine-tune a GPT-2-based generative model on a small in-domain support set for response generation.
• Integrate a retrieval component that selects the most similar dialogue context from the support set using semantic encoding.
• Encode both target dialogue contexts and support set dialogues using the same transformer encoder for semantic similarity computation.
• Use a contrastive ranking mechanism to select the best response between the generated candidate and the retrieved candidate.
• Combine generative and retrieved responses via a learned ranking module to produce the final output.
• Apply knowledge transfer from a pre-trained language model to enhance zero-shot and few-shot adaptation performance.

Experimental results

Research questions

• RQ1 Can a hybrid generative-retrieval approach outperform purely generative or retrieval-based models in few-shot dialogue domain adaptation?
• RQ2 How effective is fine-tuning a large pre-trained language model like GPT-2 on a small in-domain support set for response generation?
• RQ3 To what extent does retrieval-based fallback improve response diversity and consistency in low-resource settings?
• RQ4 Can such a hybrid model generalize effectively to an unseen multi-domain benchmark like MultiWOZ?
• RQ5 What is the human-rated performance gain of this approach over existing state-of-the-art systems in few-shot adaptation?

Key findings

• The proposed model achieved a win rate of 56.85% in human pairwise comparisons, outperforming the second-place system by over 4 percentage points.
• It achieved state-of-the-art performance on the Meta-WOZ dataset in human evaluation, with a significant margin over the baseline and other competing systems.
• The model demonstrated competitive generalization performance on the unseen MultiWOZ dataset, indicating strong zero-shot adaptation capability.
• Automatic metrics such as BLEU, METEOR, and ROUGE-L showed consistent improvements over both generative-only and retrieval-only baselines.
• The hybrid architecture improved response diversity and consistency, with retrieval serving as a robust fallback mechanism in low-confidence generation scenarios.
• The model's performance was significantly enhanced by fine-tuning on a small in-domain support set, proving effective even with limited labeled data.