[Paper Review] Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems

Asma Ghandeharioun, Judy Hanwen Shen+5 more|arXiv (Cornell University)|Jun 21, 2019

Topic ModelingComputer Science43 references51 citations

TL;DR

The paper proposes a self-play framework to approximate interactive human evaluation for open-domain dialog systems using psychology-inspired metrics, achieving high correlation with human judgments (r > 0.7, p < .05). It also regularizes hierarchical dialog models with emotion and semantics distillation and opens sources for data and platform.

ABSTRACT

Building an open-domain conversational agent is a challenging problem. Current evaluation methods, mostly post-hoc judgments of static conversation, do not capture conversation quality in a realistic interactive context. In this paper, we investigate interactive human evaluation and provide evidence for its necessity; we then introduce a novel, model-agnostic, and dataset-agnostic method to approximate it. In particular, we propose a self-play scenario where the dialog system talks to itself and we calculate a combination of proxies such as sentiment and semantic coherence on the conversation trajectory. We show that this metric is capable of capturing the human-rated quality of a dialog model better than any automated metric known to-date, achieving a significant Pearson correlation (r&gt;.7, p

Motivation & Objective

• Argue that multi-turn interactive evaluation is essential for open-domain dialog systems.
• Introduce a self-play framework to approximate interactive human evaluation using psychology-informed metrics.
• Demonstrate that a hybrid metric computed on self-play correlates highly with human judgments (r > 0.7).
• Regularize hierarchical dialog models with sentiment and semantic knowledge distillation to improve interactive performance.
• Open-source the evaluation platform and Reddit-based dataset to foster reproducibility.

Proposed method

• Define sentiment, semantics, and engagement metrics inspired by psychology (sentiment coherence, Infersent-based semantic similarity, lexical/coherence measures, engagement via questions).
• Train a function that maps these metrics to human judgments using data from interactive evaluations.
• Apply self-play by letting the dialog model generate a 10-turn trajectory and compute the same metrics, using the learned mapping to predict quality.
• Regularize top-level Context RNNs of HRED/VHRED/VHCR with EI distillation to encode sentiment and semantics in utterances.
• Compare baseline and EI variants across Cornell and Reddit datasets using interactive evaluation and static metrics.
• Open-source the code, data, and evaluation platform.

Experimental results

Research questions

• RQ1 Can interactive multi-turn evaluation be effectively approximated by a self-play framework?
• RQ2 Do psychology-inspired metrics (sentiment, semantics, engagement) predict human judgments of dialog quality?
• RQ3 Does knowledge distillation of sentiment and semantics into hierarchical dialog models improve interactive evaluation performance?
• RQ4 How does the self-play based metric compare to traditional automated metrics in correlating with human judgments?

Key findings

• Self-play based hybrid metric M_H strongly correlates with human ratings across models (r > .7, p < .05).
• EI regularization improves interactive evaluation outcomes across HRED, VHRED, and VHCR architectures on both Cornell and Reddit data.
• Static automatic metrics (perplexity, KL, embedding distances) show weaker or inconsistent correlation with human judgments.
• EI models elicit longer, more engaging, and semantically coherent conversations in interactive settings and via self-play.
• Reddit-based training data yield better interactive evaluation performance than Cornell data.
• Standalone static human evaluation shows high variance and low inter-annotator agreement, underscoring the value of interactive evaluation.