[Paper Review] Towards a Human-like Open-Domain Chatbot

Daniel Adiwardana|arXiv (Cornell University)|Jan 27, 2020

Topic Modeling56 references267 citations

TL;DR

Meena is a 2.6B-parameter end-to-end open-domain chatbot trained on 40B words from public conversations, introducing the SSA evaluation metric and showing strong correlation between perplexity and human-like behavior.

ABSTRACT

We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We also propose a human evaluation metric called Sensibleness and Specificity Average (SSA), which captures key elements of a human-like multi-turn conversation. Our experiments show strong correlation between perplexity and SSA. The fact that the best perplexity end-to-end trained Meena scores high on SSA (72% on multi-turn evaluation) suggests that a human-level SSA of 86% is potentially within reach if we can better optimize perplexity. Additionally, the full version of Meena (with a filtering mechanism and tuned decoding) scores 79% SSA, 23% higher in absolute SSA than the existing chatbots we evaluated.

Motivation & Objective

• Motivate open-domain chatbots to achieve human-like sensibleness and specificity in multi-turn conversations.
• Propose a simple, scalable human-evaluation SSA metric for assessing chatbots.
• Demonstrate that low perplexity in large end-to-end models correlates with higher SSA.
• Show that a fully tuned decoding/filters version reaches higher SSA than prior chatbots.
• Provide a practical training and decoding setup for large-scale end-to-end chatbots.

Proposed method

• Train a seq2seq model using the Evolved Transformer architecture on 40B words of filtered public social media conversations.
• Use a context window of up to 7 turns for input and predict the next response.
• Represent text with 8K BPE subwords via SentencePiece for a 341GB dataset.
• Optimize with Adafactor on TPU-v3 pods for 30 days, observing ~10T tokens.
• Decode with sample-and-rank: sample N candidates at temperature T and pick the highest-probability candidate.
• Evaluate with the SSA metric (Sensibleness and Specificity Average) via static and interactive human evaluations.

Experimental results

Research questions

• RQ1 Can an end-to-end large-scale neural model achieve human-like multi-turn open-domain conversation with low perplexity?
• RQ2 Does a simple perplexity-based objective correlate with human judgments of sensibleness and specificity?
• RQ3 Does decoding with a sample-and-rank strategy yield diverse yet high-quality responses at scale?
• RQ4 How does Meena compare to existing chatbots (Cleverbot, Mitsuku, XiaoIce, DialoGPT) on SSA and other metrics?

Key findings

• Meena achieves a test perplexity of 10.2 and attains 72% SSA for the best end-to-end model and 79% SSA with a filtering/decoding upgrade.
• SSA shows a strong correlation with perplexity (static SSA R2≈0.94, interactive SSA similarly high).
• The human upper bound on SSA is about 86%, with Meena approaching human-like performance but still below the human level.
• The full Meena with filtering/decoding scores 79% SSA, which is 23% higher in absolute SSA than the evaluated existing chatbots.
• Perplexity-based optimization can outperform handcrafted, multi-component chatbots in sensibleness and specificity on open-domain tasks.