[Paper Review] Neural Response Generation with Dynamic Vocabularies

Yu Wu, Wei Wu+4 more|arXiv (Cornell University)|Nov 30, 2017

Topic ModelingComputer Science50 citations

TL;DR

The paper introduces DVS2S, a dynamic vocabulary sequence-to-sequence model for open-domain chat, jointly learning per-input vocabularies and responses to improve quality and decoding efficiency.

ABSTRACT

We study response generation for open domain conversation in chatbots. Existing methods assume that words in responses are generated from an identical vocabulary regardless of their inputs, which not only makes them vulnerable to generic patterns and irrelevant noise, but also causes a high cost in decoding. We propose a dynamic vocabulary sequence-to-sequence (DVS2S) model which allows each input to possess their own vocabulary in decoding. In training, vocabulary construction and response generation are jointly learned by maximizing a lower bound of the true objective with a Monte Carlo sampling method. In inference, the model dynamically allocates a small vocabulary for an input with the word prediction model, and conducts decoding only with the small vocabulary. Because of the dynamic vocabulary mechanism, DVS2S eludes many generic patterns and irrelevant words in generation, and enjoys efficient decoding at the same time. Experimental results on both automatic metrics and human annotations show that DVS2S can significantly outperform state-of-the-art methods in terms of response quality, but only requires 60% decoding time compared to the most efficient baseline.

Motivation & Objective

• Motivate reducing noise and generic responses by abandoning a fixed decoding vocabulary in favor of input-specific vocabularies.
• Propose a dynamic vocabulary S2S framework that jointly learns vocabulary construction and response generation.
• Show that dynamic vocabularies improve response quality and reduce decoding time on large-scale conversation data.

Proposed method

• Encode input with a biGRU-based encoder to obtain hidden representations.
• For each input, sample a target vocabulary T from a multivariate Bernoulli distribution conditioned on the input.
• Decode responses using only the selected vocabulary T with an attention-based decoder and a projection over T.
• Formulate training as maximizing a variational lower bound and optimize via Monte Carlo sampling of T with S samples.
• Partition vocabulary into function words (always included) and content words (beta parameters depend on encoder state).
• Jointly learn word selection and response generation so decoding uses a smaller, relevant vocabulary.

Experimental results

Research questions

• RQ1 Can a per-input dynamic vocabulary improve the relevance and informativeness of open-domain responses?
• RQ2 Does joint learning of vocabulary construction and response generation yield better performance than separate training?
• RQ3 What is the impact of dynamic vocabulary size on response quality and decoding efficiency?

Key findings

• DVS2S significantly outperforms baselines on automatic metrics such as BLEU-1, Embedding Average, and Distinct-1.
• DVS2S achieves higher Distinct-1 and Distinct-2, indicating more diverse and informative responses.
• Human evaluation shows DVS2S more informative and less invalid than baselines.
• DVS2S reduces decoding time by about 40% compared with the most efficient baseline in the same setting.
• A top-content-word vocabularies of around 1,000 words cover about 80% of ground-truth words in test responses.
• Joint learning of vocabulary construction and response generation yields improvements over a separately trained variant (S-DVS2S).