[Paper Review] Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations

Karthik Gopalakrishnan, Behnam Hedayatnia+6 more|arXiv (Cornell University)|Aug 23, 2023

Topic Modeling4 citations

TL;DR

This paper introduces Topical-Chat, a human-human, knowledge-grounded open-domain conversation dataset comprising ~11,000 dialogues across 8 broad topics, with no predefined roles and asymmetric or symmetric reading sets to reflect real-world knowledge diversity. The authors train Transformer-based encoder-decoder models on this dataset, demonstrating improved response diversity and knowledge grounding through automated and human evaluations, with the best model achieving 0.84 unigram F1 and 0.83 bigram diversity on the frequent test set.

ABSTRACT

Building socialbots that can have deep, engaging open-domain conversations with humans is one of the grand challenges of artificial intelligence (AI). To this end, bots need to be able to leverage world knowledge spanning several domains effectively when conversing with humans who have their own world knowledge. Existing knowledge-grounded conversation datasets are primarily stylized with explicit roles for conversation partners. These datasets also do not explore depth or breadth of topical coverage with transitions in conversations. We introduce Topical-Chat, a knowledge-grounded human-human conversation dataset where the underlying knowledge spans 8 broad topics and conversation partners don't have explicitly defined roles, to help further research in open-domain conversational AI. We also train several state-of-the-art encoder-decoder conversational models on Topical-Chat and perform automated and human evaluation for benchmarking.

Motivation & Objective

• To address the lack of natural, knowledge-grounded open-domain conversation datasets that reflect real-world asymmetries in conversational knowledge and topic depth.
• To enable research in open-domain conversational AI by providing a dataset with broad topical coverage and smooth topic transitions.
• To benchmark state-of-the-art models on a realistic, human-generated knowledge-grounded dialogue benchmark.
• To evaluate the effectiveness of knowledge grounding in response generation using both automated and human evaluation metrics.

Proposed method

• The dataset was collected via Amazon Mechanical Turk, pairing workers who received topical reading sets (Wikipedia, Washington Post, Reddit) and engaged in natural, role-free conversations.
• Reading sets were designed to be symmetric or asymmetric to reflect real-world knowledge imbalances between conversation partners.
• Each conversation was annotated for reading set utilization, sentiment, and other dimensions to support evaluation.
• The authors fine-tuned Transformer-based encoder-decoder models on Topical-Chat, using knowledge-aware attention mechanisms with a context window size $W_H = 32$.
• Models were trained with and without pre-training on BookCorpus, using beam search (beam size 5) and byte-pair encoding (BPE) for vocabulary tokenization.
• Knowledge grounding was implemented by conditioning the decoder on both conversation history and retrieved knowledge sentences, with $W_K = 128$ to accommodate long knowledge inputs.

Experimental results

Research questions

• RQ1 How effective are Transformer-based models at generating responses grounded in diverse, real-world knowledge sources in open-domain conversations?
• RQ2 To what extent does knowledge grounding improve response coherence, topical relevance, and engagement in open-domain dialogue?
• RQ3 How does model performance vary with different training strategies, such as pre-training and knowledge integration?
• RQ4 How well do automated metrics correlate with human judgments of response quality in knowledge-grounded dialogue?

Key findings

• The best-performing model (TF with knowledge and pre-training) achieved a unigram F1 of 0.22 on the frequent test set and 0.84 bigram diversity, indicating high lexical informativeness and diversity.
• Knowledge-grounded models showed improved unigram F1 (0.22) and higher perplexity (34.1) compared to non-knowledge baselines (F1: 0.16, PPL: 29.8), confirming better alignment with ground-truth responses.
• Human evaluation showed that models using knowledge were rated to leverage knowledge more effectively (mean 0.80) than non-knowledge models (0.08), though effectiveness remained moderate.
• The model with $W_H = 32$ achieved the best automated metrics, indicating that overly large context windows impair attention to key dialog tokens.
• Human agreement on 'interesting' and 'continue conversation' labels was low (kappa < 0.3), suggesting subjective nature of these attributes, but other metrics (comprehensibility, on-topic) showed high reliability (kappa ≥ 0.67).
• The dataset supports natural topic transitions and diverse knowledge use, as evidenced by high diversity scores and annotations showing consistent reading set utilization.