[Paper Review] Topic Aware Neural Response Generation

Xing Chen, Wei Wu+5 more|arXiv (Cornell University)|Jun 21, 2016

Topic ModelingComputer Science317 citations

TL;DR

The paper introduces TA-Seq2Seq, a topic-aware sequence-to-sequence model that uses a joint attention mechanism over input messages and pre-trained topic words to generate more informative and diverse chatbot responses, outperforming state-of-the-art baselines. It biases generation toward topic words and demonstrates improvements on large-scale Chinese conversational data.

ABSTRACT

We consider incorporating topic information into the sequence-to-sequence framework to generate informative and interesting responses for chatbots. To this end, we propose a topic aware sequence-to-sequence (TA-Seq2Seq) model. The model utilizes topics to simulate prior knowledge of human that guides them to form informative and interesting responses in conversation, and leverages the topic information in generation by a joint attention mechanism and a biased generation probability. The joint attention mechanism summarizes the hidden vectors of an input message as context vectors by message attention, synthesizes topic vectors by topic attention from the topic words of the message obtained from a pre-trained LDA model, and let these vectors jointly affect the generation of words in decoding. To increase the possibility of topic words appearing in responses, the model modifies the generation probability of topic words by adding an extra probability item to bias the overall distribution. Empirical study on both automatic evaluation metrics and human annotations shows that TA-Seq2Seq can generate more informative and interesting responses, and significantly outperform the-state-of-the-art response generation models.

Motivation & Objective

• Motivate the need for more informative and engaging open-domain chatbot responses.
• Leverage topic information as prior knowledge to guide response generation.
• Integrate topic words into the Seq2Seq framework via joint attention and biased generation.
• Show that topic-aware generation improves informativeness, diversity, and relevance over baselines.
• Validate the approach with large-scale data and human judgments.

Proposed method

• Use a bidirectional GRU encoder for the input message and a topic encoder for topic word embeddings derived from a Twitter LDA model.
• Compute a joint attention context by combining message attention with topic attention, where topic attention uses the final encoder state to weight topic words.
• Generate each target word from a biased distribution that includes both standard vocabulary generation and topic-word components.
• Bias the generation probability toward topic words by adding an extra probability term dependent on the decoder state, previous word, and context.
• Train on large-scale Baidu Tieba data and evaluate with perplexity, distinct-1/2 metrics, and human judgments.
• Utilize a two-step learning approach: topic word acquisition via Twitter LDA and response generation within the TA-Seq2Seq framework.

Experimental results

Research questions

• RQ1 Can incorporating topic information into Seq2Seq generation produce more informative and interesting responses?
• RQ2 Does a joint attention mechanism that integrates message content and topic words improve response quality over standard attention?
• RQ3 Does biasing the generation toward topic words increase the appearance and usefulness of topic-related content in responses?
• RQ4 How does TA-Seq2Seq compare to state-of-the-art response generation models on automatic metrics and human judgments?

Key findings

• TA-Seq2Seq yields more informative, diverse, and topic-relevant responses than baselines.
• Topic-aware variants (TopicConcat, TopicAttention) improve human judgments compared to non-topic models, with TA-Seq2Seq achieving the best performance.
• TA-Seq2Seq attains competitive perplexity while enhancing distinct-1 and distinct-2 metrics, indicating richer content in outputs.
• Topic attention that uses the final message state (h_T) helps suppress irrelevant topic words, improving relevance.
• The biased generation component increases the likelihood of generating topic words, contributing to content richness.
• Statistical testing confirms improvements are significant (p-value < 0.01).