[Paper Review] BARTScore: Evaluating Generated Text as Text Generation

Weizhe Yuan, Graham Neubig+1 more|arXiv (Cornell University)|Jun 22, 2021

Topic ModelingComputer Science67 references318 citations

TL;DR

BARTScore treats evaluation of generated text as a text generation task using a pre-trained seq2seq model (BART), scoring hypotheses via conditional generation probabilities across multiple directions and perspectives, often outperforming existing metrics in diverse Settings.

ABSTRACT

A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e.g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets (e.g., machine translation, text summarization) and 7 different perspectives (e.g., informativeness, factuality). Code to calculate BARTScore is available at https://github.com/neulab/BARTScore, and we have released an interactive leaderboard for meta-evaluation at http://explainaboard.nlpedia.ai/leaderboard/task-meval/ on the ExplainaBoard platform, which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.

Motivation & Objective

• Motivate evaluating generated text through its generation probability to better align with pre-training objectives.
• Develop BARTScore variants to assess multiple evaluation perspectives (e.g., informativeness, factuality, fluency).
• Show that BARTScore achieves strong correlations with human judgments across MT, summarization, and data-to-text datasets.
• Investigate the impact of prompts and task-specific fine-tuning on metric performance.

Proposed method

• Define BARTScore as the weighted log-probability of a target text given a source or reference text: BARTScore = sum_t w_t log p(y_t | y_<t, x, theta).
• Provide four evaluation directions: s→h (faithfulness), r→h (precision), h→r (recall), and a combined F-score from r↔h for semantic overlap.
• Introduce variants: vanilla BARTScore, BARTScore-CNN (summarization fine-tune), BARTScore-CNN-Para (summarize then paraphrase fine-tune), and BARTScore-Prompt (prompt-based inputs/outputs).
• Apply prompt prompting by adding short phrases to inputs/outputs and ensemble over prompts for stability.
• Explore downstream fine-tuning on summarization (CNNDM) and paraphrasing (ParaBank2) to align pre-training with evaluation tasks.
• Use bootstrapping for significance testing when comparing metrics.

Experimental results

Research questions

• RQ1 Can BARTScore reliably correlate with human judgments across MT, summarization, and data-to-text tasks?
• RQ2 Do prompts and task-oriented fine-tuning improve BARTScore’s correlation with human judgments?
• RQ3 How does BARTScore perform across perspectives such as informativeness, factuality, coherence, and fluency compared to existing metrics?
• RQ4 What are the robustness and bias characteristics of BARTScore when evaluating high-quality versus lower-quality generated text?

Key findings

• BARTScore variants with task-aligned fine-tuning significantly improve correlation in several language pairs and tasks.
• Adding prompts (e.g., a phrase like “Such as”) can boost MT correlation, showing the value of prompt-augmented evaluation.
• Fine-tuning on CNNDM boosts performance for summarization and, to a degree, data-to-text tasks, while paraphrase-focused fine-tuning may hurt some scenarios.
• Prompt ensembles consistently improve semantic-overlap based perspectives (informativeness, pyramid/coverage), but have mixed effects on factuality.
• BARTScore generally shows robustness across reference lengths and top-quality outputs, though it is less effective at distinguishing extractive systems in certain summarization datasets.