[Paper Review] Personalizing Dialogue Agents: I have a dog, do you have pets too?

Saizheng Zhang, Emily Dinan+4 more|arXiv (Cornell University)|Jan 22, 2018

Topic Modeling25 references196 citations

TL;DR

This paper introduces persona-conditioned chit-chat and a large persona-chat dataset. It shows memory-augmented and retrieval-based models benefit from explicit persona conditioning for next-utterance prediction, with human evaluation favoring persona-rich data over OpenSubtitles/Twitter baselines.

ABSTRACT

Chit-chat models are known to have several problems: they lack specificity, do not display a consistent personality and are often not very captivating. In this work we present the task of making chit-chat more engaging by conditioning on profile information. We collect data and train models to (i) condition on their given profile information; and (ii) information about the person they are talking to, resulting in improved dialogues, as measured by next utterance prediction. Since (ii) is initially unknown our model is trained to engage its partner with personal topics, and we show the resulting dialogue can be used to predict profile information about the interlocutors.

Motivation & Objective

• Introduce a task to personalize chit-chat by conditioning on a configurable profile (persona).
• Create and release the persona-chat dataset with original and revised personas.
• Evaluate generative and ranking models on next utterance prediction with and without persona information.
• Show that persona conditioning improves next-utterance prediction and enables partner profiling from dialogue.

Proposed method

• Propose four modeling paradigms: Seq2Seq, Profile Memory Network, Key-Value Profile Memory Network, and standard ranking baselines (IR, StarSpace).
• Condition inputs on the agent’s own persona, the partner’s persona, or both; evaluate under three persona settings (No Persona, Original Persona, Revised Persona).
• Use perplexity (ppl), hits@1, and F1-like next-utterance classification loss as evaluation metrics.
• Construct a large crowd-sourced dataset (persona-chat) with 162,064 utterances over 10,907 dialogues, each participant given a persona.
• Utilize memory-augmented architectures to attend over persona sentences and optionally over dialogue history keys/values to influence response selection.
• Provide open-source code and ParlAI integration for easy replication.

Experimental results

Research questions

• RQ1 Does conditioning dialogue agents on explicit persona information improve next-utterance prediction compared to persona-free baselines?
• RQ2 How does conditioning on the partner’s persona affect dialogue quality and the ability to predict or recover profile information?
• RQ3 Do revised personas (word-overlap removed) pose a greater learning challenge, and can memory-attention models still leverage them effectively?
• RQ4 How do generative versus ranking models compare when conditioned on persona information in terms of fluency, engagement, and consistency?

Key findings

• Conditioning on persona information improves next-utterance prediction for both generative and ranking models.
• Memory-augmented attention over persona memories yields better performance than simple concatenation baselines, with KV-profile memories offering the strongest gains in some settings.
• Ranking models outperform generative models on the next-utterance selection task (as measured by hits@1).
• Original (non-revised) personas lead to larger gains than revised personas due to word overlap; revised personas still provide benefits when using memory-based models.
• Models trained on persona-chat data produce more engaging and consistent conversations than models trained on OpenSubtitles or Twitter data, according to human evaluations.
• Human evaluators could sometimes detect persona conditioning, yet fluency and consistency remained competitive with non-persona baselines.