[Paper Review] Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation

Lili Mou, Yiping Song+4 more|arXiv (Cornell University)|Jul 4, 2016

Topic ModelingComputer Science25 references191 citations

TL;DR

The paper introduces a content-introducing approach for generative short-text conversation that predicts a keyword via PMI and generates replies using a seq2BF model that constructs backward and forward sequences around the keyword.

ABSTRACT

Using neural networks to generate replies in human-computer dialogue systems is attracting increasing attention over the past few years. However, the performance is not satisfactory: the neural network tends to generate safe, universally relevant replies which carry little meaning. In this paper, we propose a content-introducing approach to neural network-based generative dialogue systems. We first use pointwise mutual information (PMI) to predict a noun as a keyword, reflecting the main gist of the reply. We then propose seq2BF, a "sequence to backward and forward sequences" model, which generates a reply containing the given keyword. Experimental results show that our approach significantly outperforms traditional sequence-to-sequence models in terms of human evaluation and the entropy measure, and that the predicted keyword can appear at an appropriate position in the reply.

Motivation & Objective

• Motivate the need to overcome universal, dull replies in neural dialogue systems.
• Propose a two-step approach: (i) keyword (noun) prediction via PMI to capture the reply gist, (ii) seq2BF to generate a fluent reply containing the keyword.
• Demonstrate that incorporating a content keyword improves human evaluation and information content over traditional seq2seq models.

Proposed method

• Predict a noun keyword for the reply using PMI against the query words.
• Use a seq2BF model to generate a reply by first producing a backward sequence from the keyword, then a forward sequence conditioned on the keyword and query.
• Implement seq2BF as an asyn-B/F variant with separate backward and forward seq2seq components.
• Train the backward generator with reversed halves of randomly split reply sequences to learn constrained generation.
• Train the forward generator as a standard seq2seq encoder-decoder conditioned on the partial reply and the keyword.

Experimental results

Research questions

• RQ1 Can PMI-based keyword prediction produce content-rich keywords that improve dialogue relevance?
• RQ2 Does the seq2BF mechanism allow the predicted keyword to appear at arbitrary positions while maintaining fluency?
• RQ3 Do content-introducing seq2BF models outperform pure seq2seq and non-content introducing seq2BF models in human evaluation and entropy-based metrics?

Key findings

• Content-introducing seq2BF yields higher pointwise human evaluation scores than seq2seq and seq2BF without keyword content introduction.
• seq2BF with content introduction increases entropy of replies by about 30% relative to seq2BF without keywords, reflecting richer information.
• The keyword-containing seq2BF model longer, more informative replies outperform baselines in pairwise human evaluations.
• Case studies show the keyword can be placed at various positions in the reply without sacrificing fluency.
• Intrinsic metrics indicate seq2BF+ produces richer content (higher entropy) while maintaining reasonable length compared to baselines.