[Paper Review] News Summarization and Evaluation in the Era of GPT-3

Tanya Goyal, Junyi Jessy Li+1 more|arXiv (Cornell University)|Sep 26, 2022

Topic Modeling181 citations

TL;DR

This paper compares GPT-3 prompted summaries against fine-tuned models on news datasets, showing humans prefer GPT-3 outputs, while standard automatic metrics struggle to evaluate them; it also explores keyword/aspect summarization and releases a large generated-summaries corpus and human judgments.

ABSTRACT

The recent success of prompting large language models like GPT-3 has led to a paradigm shift in NLP research. In this paper, we study its impact on text summarization, focusing on the classic benchmark domain of news summarization. First, we investigate how GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, prompted using only a task description, but these also do not suffer from common dataset-specific issues such as poor factuality. Next, we study what this means for evaluation, particularly the role of gold standard test sets. Our experiments show that both reference-based and reference-free automatic metrics cannot reliably evaluate GPT-3 summaries. Finally, we evaluate models on a setting beyond generic summarization, specifically keyword-based summarization, and show how dominant fine-tuning approaches compare to prompting. To support further research, we release: (a) a corpus of 10K generated summaries from fine-tuned and prompt-based models across 4 standard summarization benchmarks, (b) 1K human preference judgments comparing different systems for generic- and keyword-based summarization.

Motivation & Objective

• Assess how prompt-based GPT-3 summaries stack up against state-of-the-art fine-tuned models on standard news summarization benchmarks.
• Investigate the reliability of reference-based and reference-free automatic evaluation metrics for GPT-3 prompted summaries.
• Examine GPT-3 prompted summaries in specialized settings beyond generic summarization, including keyword-based and aspect-based summarization.
• Provide supporting resources including a large corpus of generated summaries and human preferences to facilitate future research.

Proposed method

• Compare GPT-3 prompted summaries (text-davinci-002) with BRIO fine-tuned models and T0 prompts using a human A/B study on CNN/DM and BBC-based data.
• Tune prompt length to match dataset styles (CNN/DM vs XSum) to ensure fair comparison in length and style.
• Evaluate automatic metrics (ROUGE, BLEU, METEOR, BERTScore, MoverScore, QA-based metrics) against human judgments for GPT-3 prompted vs fine-tuned summaries.
• Assess reference-free metrics (SUPERT, BLANC, QuestEval, QAFactEval, FactCC, DAE, SummaC, etc.) for their ability to align with human preferences.
• Explore beyond generic summarization with keyword-focused and aspect-focused prompts, comparing GPT-3-D2 against CTRLSum baselines.

Experimental results

Research questions

• RQ1 How do prompt-based GPT-3 summaries compare to state-of-the-art fine-tuned models in overall quality and style preferences by humans?
• RQ2 Do standard automatic evaluation metrics reliably evaluate GPT-3 prompted summaries?
• RQ3 Can prompting approaches extend effectively to keyword-based and aspect-based summarization tasks?
• RQ4 What resources (summaries and human judgments) can support future research in GPT-3-based summarization?

Key findings

• Humans overwhelmingly prefer GPT-3 prompted summaries over fine-tuned models across CNN/DM and BBC-style prompts.
• GPT-3 prompted summaries achieve higher human preference despite scoring worse on traditional automatic metrics like ROUGE and BLEU.
• Reference-based automatic metrics fail to reliably evaluate GPT-3 prompted summaries, often underestimating quality relative to human judgments.
• Reference-free metrics also fail to consistently track human preferences for GPT-3 prompted outputs, with factuality and quality signals not aligning well across datasets.
• GPT-3 prompts yield strong keyword-based summarization performance but show mixed results for aspect-based summarization, highlighting limits of simple prompts for high-level aspects.
• A new resource release provides 10K generated summaries and 1K human preference judgments across 4 benchmarks to support further research.