[Paper Review] Incremental Transformer with Deliberation Decoder for Document Grounded Conversations

Zekang Li, Cheng Niu+4 more|arXiv (Cornell University)|Jul 20, 2019

Topic Modeling26 references18 citations

TL;DR

This paper proposes an Incremental Transformer with Deliberation Decoder for document-grounded dialogue systems, enhancing context coherence and knowledge relevance by incrementally encoding multi-turn utterances with document knowledge and using a two-pass decoder that first ensures coherence and then refines responses with document facts. The model significantly outperforms state-of-the-art baselines in both automatic and manual evaluations.

ABSTRACT

Document Grounded Conversations is a task to generate dialogue responses when chatting about the content of a given document. Obviously, document knowledge plays a critical role in Document Grounded Conversations, while existing dialogue models do not exploit this kind of knowledge effectively enough. In this paper, we propose a novel Transformer-based architecture for multi-turn document grounded conversations. In particular, we devise an Incremental Transformer to encode multi-turn utterances along with knowledge in related documents. Motivated by the human cognitive process, we design a two-pass decoder (Deliberation Decoder) to improve context coherence and knowledge correctness. Our empirical study on a real-world Document Grounded Dataset proves that responses generated by our model significantly outperform competitive baselines on both context coherence and knowledge relevance.

Motivation & Objective

• To address the challenge of effectively integrating document-level knowledge into multi-turn dialogue systems.
• To improve response coherence and knowledge relevance in document-grounded conversations beyond existing end-to-end models.
• To model the human-like cognitive process of drafting a response and then refining it using background knowledge.
• To design a unified representation that combines multi-turn dialogue history with relevant document content.
• To develop a two-pass decoding mechanism that separately optimizes for context coherence and knowledge correctness.

Proposed method

• An Incremental Transformer is designed to encode multi-turn utterances and document knowledge in a sequential, context-aware manner using self- and cross-attention mechanisms.
• The Deliberation Decoder employs a two-pass architecture: the first pass generates a draft response focused on coherence with the dialogue history.
• The second pass refines the first-pass output by attending to relevant document knowledge, improving factual accuracy and relevance.
• The second-pass decoder directly fine-tunes the first-pass result using both the dialogue context and document representations.
• The model uses cross-attention to align utterances with relevant document passages and integrates knowledge dynamically during decoding.
• The incremental encoding scheme allows the model to build context representations step-by-step, maintaining long-range dependencies across turns.

Experimental results

Research questions

• RQ1 Can a two-pass decoder architecture improve both context coherence and knowledge relevance in document-grounded dialogue generation?
• RQ2 How effective is incremental encoding in capturing multi-turn context while integrating document knowledge?
• RQ3 Does imitating the human cognitive process of drafting and refining responses lead to better dialogue quality?
• RQ4 To what extent does the proposed model outperform existing Transformer-based baselines on document-grounded dialogue benchmarks?
• RQ5 Can the model guide future dialogue turns by incorporating relevant knowledge proactively?

Key findings

• The proposed model achieves significant improvements in response coherence and knowledge relevance compared to competitive baselines on the Document Grounded Conversations dataset.
• Manual evaluation confirms that responses generated by the model are more contextually consistent and better grounded in document facts.
• The two-pass decoding mechanism effectively reduces knowledge errors, as shown in case studies where the second pass corrects factual inaccuracies from the first pass.
• The second-pass decoder enhances response quality by incorporating more detailed and relevant document information than the first pass.
• In some cases, the second-pass response guides future dialogue by asking knowledge-grounded follow-up questions, indicating better conversational awareness.
• The model demonstrates strong generalization, with responses that are not only accurate but also naturally flowing and contextually appropriate.