[Paper Review] Wizard of Wikipedia: Knowledge-Powered Conversational agents

Emily Dinan, Stephen Roller+4 more|arXiv (Cornell University)|Nov 3, 2018

Topic Modeling22 references479 citations

TL;DR

The paper introduces Transformer Memory Network architectures that retrieve, read, and ground open-domain dialogues in Wikipedia knowledge, and releases a large dataset of knowledge-grounded conversations for training and evaluation.

ABSTRACT

In open-domain dialogue intelligent agents should exhibit the use of knowledge, however there are few convincing demonstrations of this to date. The most popular sequence to sequence models typically "generate and hope" generic utterances that can be memorized in the weights of the model when mapping from input utterance(s) to output, rather than employing recalled knowledge as context. Use of knowledge has so far proved difficult, in part because of the lack of a supervised learning benchmark task which exhibits knowledgeable open dialogue with clear grounding. To that end we collect and release a large dataset with conversations directly grounded with knowledge retrieved from Wikipedia. We then design architectures capable of retrieving knowledge, reading and conditioning on it, and finally generating natural responses. Our best performing dialogue models are able to conduct knowledgeable discussions on open-domain topics as evaluated by automatic metrics and human evaluations, while our new benchmark allows for measuring further improvements in this important research direction.

Motivation & Objective

• Motivate and study open-domain dialogue that can recall and ground knowledge from large text sources.
• Create a large, publicly available dataset of knowledge-grounded conversations linked to Wikipedia.
• Develop architectures that retrieve, read, and condition on retrieved knowledge to generate engaging responses.
• Evaluate both automatic metrics and human judgments to assess knowledge grounding and engagingness.

Proposed method

• Use an information retrieval step to fetch a small set of candidate knowledge passages from Wikipedia based on the topic and dialogue history.
• Encode knowledge sentences and dialogue context with Transformer encoders and attend over memory to form a context-aware representation.
• Provide retrieval-based and generative dialogue models (Retrieval Transformer Memory Network and Generative Transformer Memory Network) that select knowledge and produce responses.
• In two-stage variants, separate knowledge selection and response generation components; in end-to-end variants, jointly encode knowledge and dialogue for generation.
• Apply knowledge dropout to improve robustness when knowledge selection is imperfect.
• Pretrain components on large corpora (e.g., Reddit) and optionally fine-tune on SQuAD-like tasks to improve retrieval/grounding performance.

Experimental results

Research questions

• RQ1 Can a knowledge-grounded dialogue model effectively retrieve and ground on Wikipedia passages to produce engaging responses?
• RQ2 How do retrieval-based and generative Transformer Memory Network architectures compare in terms of knowledge grounding and conversational quality?
• RQ3 What is the impact of explicit knowledge supervision and knowledge dropout on grounding and generation?
• RQ4 To what extent does a large, publicly released Wizard of Wikipedia dataset enable improvements in knowledge-grounded open-domain dialogue?
• RQ5 How does model performance differ on seen versus unseen topics or knowledge?

Key findings

• Retrieval-based models consistently improve knowledge grounding and engagingness versus baselines, with memory-augmented transformers achieving strong Recall@1 and Wiki F1 in human evaluations.
• Generative models conditioned on knowledge outperform baselines lacking knowledge, especially when gold knowledge is available; end-to-end variants can outperform two-stage variants on some metrics.
• Knowledge supervision and knowledge dropout improve robustness and overall performance, with two-stage models benefiting from strong knowledge selection modules.
• Human evaluations show retrieval-based models scoring higher on engagingness, while generative models with knowledge yield higher Wiki F1 (knowledge overlap with Wikipedia).
• The Wizard of Wikipedia dataset (22,311 dialogues, 201,999 turns) enables robust training and evaluation of knowledge-grounded dialogue systems.