[Paper Review] Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation

Shikhar Sharma, Layla El Asri+2 more|arXiv (Cornell University)|Jun 29, 2017

Topic Modeling19 references183 citations

TL;DR

The paper empirically evaluates how unsupervised automatic metrics (BLEU, METEOR, ROUGE, embedding-based metrics) correlate with human judgments in task-oriented dialogue NLG, finding METEOR generally most aligned and that multiple references improve correlation; datasets DSTC2 and Restaurants are used with several NLG models.

ABSTRACT

Automated metrics such as BLEU are widely used in the machine translation literature. They have also been used recently in the dialogue community for evaluating dialogue response generation. However, previous work in dialogue response generation has shown that these metrics do not correlate strongly with human judgment in the non task-oriented dialogue setting. Task-oriented dialogue responses are expressed on narrower domains and exhibit lower diversity. It is thus reasonable to think that these automated metrics would correlate well with human judgment in the task-oriented setting where the generation task consists of translating dialogue acts into a sentence. We conduct an empirical study to confirm whether this is the case. Our findings indicate that these automated metrics have stronger correlation with human judgments in the task-oriented setting compared to what has been observed in the non task-oriented setting. We also observe that these metrics correlate even better for datasets which provide multiple ground truth reference sentences. In addition, we show that some of the currently available corpora for task-oriented language generation can be solved with simple models and advocate for more challenging datasets.

Motivation & Objective

• Evaluate whether unsupervised automated metrics correlate with human judgments in task-oriented dialogue generation.
• Compare word-overlap and embedding-based metrics on two task-oriented datasets.
• Assess how model complexity and dataset characteristics affect metric-human alignment.

Proposed method

• Survey and compute correlations between automated metrics (BLEU, METEOR, ROUGE, Skip-Thought, embedding averages, vector extrema, greedy matching) and human judgments on DSTC2 and Restaurants datasets.
• Implement and compare several NLG models (Random, LSTM, delex-scLSTM, hierarchical-lex-delex-scLSTM) trained to translate dialogue acts into natural language.
• Use beam search decoding with slot error rate penalties to generate outputs for fair comparison.

Experimental results

Research questions

• RQ1 Do unsupervised automated metrics correlate with human judgments in task-oriented dialogue NLG as they do in non-task-oriented settings?
• RQ2 Which automated metrics best correlate with human evaluations in this domain?
• RQ3 Does having multiple reference sentences improve the correlation between automated metrics and human judgments?
• RQ4 Are complex neural decoding architectures necessary to achieve high metric scores on task-oriented NLG benchmarks?
• RQ5 Are task-oriented datasets like DSTC2 and Restaurants sufficiently challenging for current NLG models and metrics?

Key findings

• Automated metrics show positive correlation with human judgments in the task-oriented setting, unlike some non-task-oriented findings.
• METEOR consistently correlates best with human evaluations across both datasets.
• Embedding-based sentence similarity metrics yield comparable correlations to word-overlap metrics for most models.
• Multiple references (as in Restaurants) increase correlation between automated metrics and human judgments.
• Simple models (e.g., LSTM with beam search) achieve high automated-metric scores, suggesting these datasets may be less challenging and calling for larger, more complex benchmarks.