[Paper Review] Efficient Retrieval Augmented Generation from Unstructured Knowledge for Task-Oriented Dialog

David Thulke, Nico Daheim+2 more|arXiv (Cornell University)|Feb 9, 2021

Topic Modeling22 references29 citations

TL;DR

The paper investigates two efficient retrieval strategies for knowledge selection from unstructured documents in task-oriented dialogs and combines them with retrieval-augmented generation to improve response quality while reducing computation time.

ABSTRACT

This paper summarizes our work on the first track of the ninth Dialog System Technology Challenge (DSTC 9), "Beyond Domain APIs: Task-oriented Conversational Modeling with Unstructured Knowledge Access". The goal of the task is to generate responses to user turns in a task-oriented dialog that require knowledge from unstructured documents. The task is divided into three subtasks: detection, selection and generation. In order to be compute efficient, we formulate the selection problem in terms of hierarchical classification steps. We achieve our best results with this model. Alternatively, we employ siamese sequence embedding models, referred to as Dense Knowledge Retrieval, to retrieve relevant documents. This method further reduces the computation time by a factor of more than 100x at the cost of degradation in R@1 of 5-6% compared to the first model. Then for either approach, we use Retrieval Augmented Generation to generate responses based on multiple selected snippets and we show how the method can be used to fine-tune trained embeddings.

Motivation & Objective

• Motivate using unstructured knowledge to answer user turns in task-oriented dialogs beyond API-based databases.
• Develop efficient knowledge retrieval methods to handle large sets of documents within real-time constraints.
• Investigate the impact of retrieval-augmented generation that conditions responses on multiple relevant snippets.
• Evaluate hierarchical selection and dense knowledge retrieval as scalable alternatives to baseline single-stage retrieval.

Proposed method

• Formulate knowledge selection as a hierarchical classification problem to reduce computation versus single-stage retrieval.
• Propose Dense Knowledge Retrieval (DKR) using siameseRoBERTa encoders to obtain dense, similarity-based document rankings.
• Introduce Retrieval Augmented Generation (RAG) to generate responses conditioned on multiple retrieved snippets via a latent variable k.
• Explore multi-task and joint training approaches to improve detection, selection, and generation tasks.
• Compare decoding strategies (beam search, nucleus sampling) and context-length effects on generation quality.
• Evaluate with MultiWOZ-derived DSTC9 Track 1 data, including domain/entity-specific snippets and a separate attraction domain.

Experimental results

Research questions

• RQ1 How can knowledge selection from large unstructured knowledge bases be made efficient for task-oriented dialog without sacrificing accuracy?
• RQ2 What is the trade-off between hierarchical selection and dense embedding retrieval in terms of speed and retrieval quality?
• RQ3 Does retrieval-augmented generation improve responsiveness and factual correctness when multiple snippets are available?
• RQ4 Can joint or multi-task learning improve performance across detection, selection, and generation subtasks?
• RQ5 How do decoding strategies and dialog-context length affect generation quality under knowledge retrieval paradigms?

Key findings

• Hierarchical Selection achieves strong retrieval accuracy with a 20x speedup over baseline selection, enabling faster real-time responses.
• Dense Knowledge Retrieval (DKR) provides >100x speedup over hierarchical selection (and >2,500x over baseline) with competitive validation metrics but weaker generalization to new domains.
• Retrieval Augmented Generation (RAG) allows conditioning generation on multiple snippets, enabling factual corrections and more informative responses in some examples.
• Joint training with RAG improves some metrics, but gains are mixed across datasets and tasks; multi-task training yields limited overall benefits.
• Longer dialog context (up to 384 tokens) improves R@1 substantially over shorter contexts, highlighting the importance of context for knowledge retrieval.