[Paper Review] Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models

Wangchunshu Zhou, Ke Xu|arXiv (Cornell University)|Feb 12, 2020

Topic Modeling28 references4 citations

TL;DR

This paper proposes a self-supervised 'learning to compare' framework that fine-tunes BERT to evaluate natural language generation (NLG) models via pairwise comparison of generated responses, improving correlation with human preference. It uses a skill rating system for model-level evaluation and demonstrates superior performance over BLEU and perplexity in both training and evaluation, especially when fine-tuned with minimal human preference data.

ABSTRACT

Automated evaluation of open domain natural language generation (NLG) models remains a challenge and widely used metrics such as BLEU and Perplexity can be misleading in some cases. In our paper, we propose to evaluate natural language generation models by learning to compare a pair of generated sentences by fine-tuning BERT, which has been shown to have good natural language understanding ability. We also propose to evaluate the model-level quality of NLG models with sample-level comparison results with skill rating system. While able to be trained in a fully self-supervised fashion, our model can be further fine-tuned with a little amount of human preference annotation to better imitate human judgment. In addition to evaluating trained models, we propose to apply our model as a performance indicator during training for better hyperparameter tuning and early-stopping. We evaluate our approach on both story generation and chit-chat dialogue response generation. Experimental results show that our model correlates better with human preference compared with previous automated evaluation approaches. Training with the proposed metric yields better performance in human evaluation, which further demonstrates the effectiveness of the proposed model.

Motivation & Objective

• To address the poor correlation of standard metrics like BLEU and perplexity with human judgment in open-domain NLG tasks.
• To develop a self-supervised, transfer-learning-based method for automated evaluation that better reflects human preferences.
• To enable effective model-level evaluation using a skill rating system based on sample-level pairwise comparisons.
• To improve hyperparameter tuning and early stopping during training by replacing suboptimal metrics like BLEU with the proposed comparative evaluator.
• To reduce reliance on expensive human annotations by enabling effective pretraining with weak supervision and limited human preference data.

Proposed method

• Fine-tune BERT to classify which of two generated NLG outputs is better, using a pairwise classification head.
• Train the model in a self-supervised manner using strong supervision (gold references as better than generated outputs) and weak supervision (improving model quality over training steps).
• Incorporate a 'tie' option to allow the model to express uncertainty when two outputs are similarly rated.
• Use a skill rating system (similar to ELO/Trueskill) to rank NLG models based on pairwise comparison outcomes, enabling efficient model-level evaluation.
• Fine-tune the model with minimal human preference annotations to better align with human judgment.
• Apply the trained comparative evaluator as a loss or metric during training to guide hyperparameter tuning and early stopping.

Experimental results

Research questions

• RQ1 Can a self-supervised pairwise comparison model based on BERT achieve better correlation with human preference than traditional metrics like BLEU and perplexity?
• RQ2 How effective is the proposed method in improving model training when used as a training objective or early-stopping criterion?
• RQ3 What is the contribution of each component—strong supervision, weak supervision, human preference data, and the tie option—to the final performance?
• RQ4 Can the model generalize well without human preference annotations, and how does it compare to baselines in such settings?
• RQ5 Does the skill rating system provide a more accurate and efficient method for model-level evaluation than direct comparison averaging?

Key findings

• The proposed model achieves a Spearman correlation of 0.764 and Pearson correlation of 0.783 with human preference on the Dailydialog dataset, significantly outperforming BLEU and other baselines.
• Removing the pairwise comparison mechanism (w/o comparison) reduces correlation to 0.491 (Spearman), showing that comparison is essential for performance.
• The tie option improves performance, with correlation dropping to 0.557 when excluded, indicating it helps model uncertainty and generalization.
• Fine-tuning with human preference annotations yields the highest correlation, but the model still performs reasonably well (0.602 Spearman) without it, proving utility in low-resource settings.
• Using BERT as the base model is critical—training from scratch yields only 0.644 Spearman, showing the importance of pre-trained language understanding.
• When used during training, the proposed metric leads to better human-evaluated model performance, demonstrating its effectiveness in guiding hyperparameter tuning and early stopping.