[Paper Review] Learning to Retrieve Entity-Aware Knowledge and Generate Responses with Copy Mechanism for Task-Oriented Dialogue Systems

Chao-Hong Tan, Xiaoyu Yang+8 more|arXiv (Cornell University)|Dec 22, 2020

Topic Modeling30 references4 citations

TL;DR

This paper proposes a task-oriented dialogue system that leverages pre-trained language models (ELECTRA and RoBERTa) for entity-aware knowledge retrieval and response generation with a copy mechanism. It achieves second place in objective metrics and fourth in human evaluation on the DSTC9 Track 1 benchmark, demonstrating improved knowledge grounding and response fluency through latent variable modeling and post-processing strategies.

ABSTRACT

Task-oriented conversational modeling with unstructured knowledge access, as track 1 of the 9th Dialogue System Technology Challenges (DSTC 9), requests to build a system to generate response given dialogue history and knowledge access. This challenge can be separated into three subtasks, (1) knowledge-seeking turn detection, (2) knowledge selection, and (3) knowledge-grounded response generation. We use pre-trained language models, ELECTRA and RoBERTa, as our base encoder for different subtasks. For subtask 1 and 2, the coarse-grained information like domain and entity are used to enhance knowledge usage. For subtask 3, we use a latent variable to encode dialog history and selected knowledge better and generate responses combined with copy mechanism. Meanwhile, some useful post-processing strategies are performed on the model's final output to make further knowledge usage in the generation task. As shown in released evaluation results, our proposed system ranks second under objective metrics and ranks fourth under human metrics.

Motivation & Objective

• To address the challenge of task-oriented dialogue systems with unstructured knowledge access, where traditional APIs or databases are insufficient.
• To improve response generation by integrating relevant knowledge snippets retrieved from unstructured sources like FAQs and reviews.
• To enhance model generalization on out-of-domain and unseen entities by incorporating domain and entity information into the input representation.
• To generate more accurate and natural responses by combining sequence-to-sequence modeling with a knowledge copy mechanism and latent variable encoding.
• To optimize model output through post-processing strategies that improve both automatic and human evaluation scores.

Proposed method

• Utilizes ELECTRA as the base encoder for knowledge-seeking turn detection, fine-tuned on dialogue history with added domain and entity labels.
• Employs a retrieval-and-ranking pipeline using RoBERTa to select the most relevant knowledge snippets from a knowledge base based on dialogue context.
• Applies a latent variable model to jointly encode dialogue history and selected knowledge, enabling better context representation for response generation.
• Integrates a copy mechanism into the decoder to allow direct copying of knowledge content into the response, improving factual consistency.
• Uses a two-stage decoding strategy (SRG and FFBS) with beam search to generate diverse and high-quality responses.
• Applies post-processing techniques to refine model outputs, enhancing knowledge usage and human evaluation scores.

Experimental results

Research questions

• RQ1 Can pre-trained language models like ELECTRA and RoBERTa effectively detect knowledge-seeking turns in task-oriented dialogues?
• RQ2 How does incorporating domain and entity information improve knowledge selection performance in zero-shot or few-shot settings?
• RQ3 To what extent does a latent variable representation improve the alignment between dialogue history, knowledge, and generated responses?
• RQ4 How effective is the copy mechanism in preserving factual knowledge from unstructured sources in generated responses?
• RQ5 Can post-processing strategies enhance response quality without degrading automatic metrics?

Key findings

• The proposed system achieved second place in objective metrics and fourth in human evaluation on the DSTC9 Track 1 test set.
• The knowledge-aware ELECTRA model improved F1-score by 2.6% over the original ELECTRA on knowledge-seeking turn detection.
• The knowledge-copy mechanism improved BLEU-4 by 0.5%, METEOR by 1.6%, and ROUGE-L by 1.2% compared to baseline generation.
• Post-processing increased human evaluation scores for appropriateness and accuracy, though it slightly reduced automatic metrics.
• Data augmentation improved recall@1, especially in domain and entity prediction, indicating better generalization to out-of-domain examples.
• The ensemble model outperformed the baseline by a large margin in knowledge selection, with significant gains in mrr@5 and recall@5.