[Paper Review] Aiming to Know You Better Perhaps Makes Me a More Engaging Dialogue Partner

Yury Zemlyanskiy, Fei Sha|arXiv (Cornell University)|Aug 21, 2018

Topic Modeling26 references3 citations

TL;DR

This paper proposes DiscoveryScore, a metric that quantifies how much a dialogue agent learns about its human interlocutor during conversation, framing engagement as information discovery. By re-ranking responses to maximize expected DiscoveryScore using a simulated human model, the agent generates more inquisitive, personalized, and engaging dialogues, significantly outperforming baselines in human evaluations.

ABSTRACT

There have been several attempts to define a plausible motivation for a chit-chat dialogue agent that can lead to engaging conversations. In this work, we explore a new direction where the agent specifically focuses on discovering information about its interlocutor. We formalize this approach by defining a quantitative metric. We propose an algorithm for the agent to maximize it. We validate the idea with human evaluation where our system outperforms various baselines. We demonstrate that the metric indeed correlates with the human judgments of engagingness.

Motivation & Objective

• To address the lack of explicit optimization objectives for engaging chit-chat dialogue systems.
• To formalize engagement as the agent's goal of discovering personal facts about the human interlocutor.
• To develop a measurable metric—DiscoveryScore—that correlates with human perceptions of engagingness.
• To design a response selection strategy that maximizes information discovery while maintaining coherence.
• To validate the approach through human evaluation on Amazon Mechanical Turk, showing improved engagement scores.

Proposed method

• Proposes DiscoveryScore as a quantitative metric measuring the number of true personal facts discovered about the human interlocutor during a dialogue.
• Employs a personalized dialogue model (ProfileMemory+) as a simulator of the human interlocutor to estimate expected DiscoveryScore for candidate responses.
• Uses a two-step re-ranking process: first generating 100 response candidates, then simulating 10 dialogues per candidate with the ProfileMemory+ model to estimate expected DiscoveryScore.
• Applies beam search (100 beams) and greedy decoding for response generation, with DiscoveryScore-based re-ranking to select the most informative response.
• Trains the ProfileMemory+ model on both PersonaChat and OpenSubtitles to improve persona understanding and response consistency.
• Uses human evaluations on Mechanical Turk to assess fluency, consistency, and especially engagingness on a 1–5 scale, alongside persona detection accuracy.

Experimental results

Research questions

• RQ1 Can a dialogue agent’s engagingness be improved by explicitly aiming to discover personal facts about its interlocutor?
• RQ2 Does the proposed DiscoveryScore metric correlate with human judgments of engagingness in open-domain dialogue?
• RQ3 Can a response selection strategy based on maximizing expected DiscoveryScore lead to more engaging conversations than standard baselines?
• RQ4 How does the use of a simulated human model (ProfileMemory+) affect the estimation of DiscoveryScore and the resulting dialogue quality?
• RQ5 To what extent does re-ranking based on DiscoveryScore reduce self-revelation while increasing interlocutor-focused questioning?

Key findings

• The DiscoveryScore-based re-ranking method significantly improved human-rated engagingness scores across all chatbot models tested, with evaluators frequently describing the agent as 'genuinely interested'.
• There was no strong correlation between engagingness and simple metrics like response length or number of questions, but a strong correlation was found with DiscoveryScore.
• The ProfileMemory+ model achieved state-of-the-art perplexity on the PersonaChat dataset and demonstrated better persona utilization than standard seq2seq models.
• Persona Detection scores decreased slightly when using DiscoveryScore-based re-ranking, indicating reduced self-revelation and increased focus on the human’s facts.
• Human evaluators rated the DiscoveryScore-optimized system significantly higher in engagingness (p < 0.01) compared to baseline models without re-ranking.
• Example dialogues showed that the re-ranked model generated more inquisitive, contextually relevant questions, while maintaining coherence and fluency.