[Paper Review] How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

Chia‐Wei Liu, Ryan Lowe+4 more|arXiv (Cornell University)|Mar 25, 2016

Topic Modeling41 references614 citations

TL;DR

The paper shows that common unsupervised automatic metrics (BLEU, METEOR, ROUGE, and embedding-based scores) poorly correlate with human judgments for dialogue responses, across Twitter and Ubuntu domains, and argues for developing better evaluation metrics.

ABSTRACT

We investigate evaluation metrics for dialogue response generation systems where supervised labels, such as task completion, are not available. Recent works in response generation have adopted metrics from machine translation to compare a model's generated response to a single target response. We show that these metrics correlate very weakly with human judgements in the non-technical Twitter domain, and not at all in the technical Ubuntu domain. We provide quantitative and qualitative results highlighting specific weaknesses in existing metrics, and provide recommendations for future development of better automatic evaluation metrics for dialogue systems.

Motivation & Objective

• Evaluate the effectiveness of unsupervised automatic metrics for dialogue response generation without supervised task labels.
• Examine word-overlap and embedding-based metrics as predictors of human judgments.
• Compare metric performance across domain-diverse dialogue datasets (Twitter and Ubuntu).
• Provide qualitative and statistical analyses to identify weaknesses of current metrics.
• Offer recommendations for future metric design that better align with human judgments.

Proposed method

• Collect responses from diverse dialogue models (retrieval and generative) across Twitter and Ubuntu corpora.
• Evaluate proposed responses against ground-truth responses using word-overlap metrics (BLEU, METEOR, ROUGE) and embedding-based metrics (Greedy Matching, Embedding Average, Vector Extrema).
• Correlate automatic metric scores with human judgments from a controlled user study using Pearson and Spearman correlations.
• Perform qualitative analyses on example cases where metrics and humans disagree to identify failure modes.
• Analyze sensitivity of metrics to stopwords/punctuation and to differences in response length.
• Discuss limitations and provide guidelines for future metric design.

Experimental results

Research questions

• RQ1 Do standard unsupervised metrics correlate with human judgments of dialogue response quality across different domains?
• RQ2 How do word-overlap and embedding-based metrics perform in correlating with human judgments for Twitter and Ubuntu dialogue datasets?
• RQ3 What are the main shortcomings of current automatic evaluation metrics for dialogue systems?
• RQ4 What directions could yield metrics that align more closely with human judgments?

Key findings

• BLEU scores (including BLEU-4) show very weak or no correlation with human judgments in both domains, with some slight positive correlation only in Twitter.
• Embedding-based metrics distinguish model quality (state-of-the-art vs baseline) but correlate weakly or not at all with human judgments, especially on Ubuntu.
• BLEU-2 and embedding-average provide the best but still weak correlations, and correlations deteriorate when stopwords/punctuation are removed or when response lengths differ significantly.
• Qualitative examples reveal that differences in wording can mislead embedding-based metrics, and context-sensitive evaluation is needed to capture semantic adequacy.
• BLEU-3/4 often yield near-zero scores, making them unreliable for dialogue evaluation in practice.
• The paper concludes that current metrics are inadequate for unsupervised dialogue evaluation and calls for new metrics that better reflect human judgments.