[Paper Review] Comparing Abstractive Summaries Generated by ChatGPT to Real Summaries Through Blinded Reviewers and Text Classification Algorithms

Mayank Soni, Vincent Wade|arXiv (Cornell University)|Mar 30, 2023

Topic Modeling9 citations

TL;DR

The paper evaluates ChatGPT-generated abstractive summaries against real summaries using automated metrics, blinded human evaluation, and text classifiers, finding humans struggle to distinguish the two while classifiers can differentiate with high accuracy.

ABSTRACT

Large Language Models (LLMs) have gathered significant attention due to their impressive performance on a variety of tasks. ChatGPT, developed by OpenAI, is a recent addition to the family of language models and is being called a disruptive technology by a few, owing to its human-like text-generation capabilities. Although, many anecdotal examples across the internet have evaluated ChatGPT's strength and weakness, only a few systematic research studies exist. To contribute to the body of literature of systematic research on ChatGPT, we evaluate the performance of ChatGPT on Abstractive Summarization by the means of automated metrics and blinded human reviewers. We also build automatic text classifiers to detect ChatGPT generated summaries. We found that while text classification algorithms can distinguish between real and generated summaries, humans are unable to distinguish between real summaries and those produced by ChatGPT.

Motivation & Objective

• Evaluate ChatGPT's abstractive summarization against real summaries using automated metrics (ROUGE, METEOR).
• Assess whether blinded human reviewers can distinguish between ChatGPT-generated and real summaries.
• Develop and evaluate text classifiers to detect ChatGPT-generated summaries.

Proposed method

• Create a dataset of 50 CNN/Daily News summaries from Nallapati et al. (2016).
• Prompt ChatGPT with carefully chosen prompts to generate abstractive summaries for the same 50 articles.
• Compute automated metrics (ROUGE-1/2/L, ROUGE-LSUM, METEOR) between real and generated summaries.
• Conduct blinded human evaluation with two native English reviewers on 50 pairs.
• Fine-tune DistillBERT for ChatGPT vs. human summary detection and compare with Sentence Embeddings + XGBoost.

Figure 1: User Interface of Summaries Shown to Human Reviewers

Experimental results

Research questions

• RQ1 Can ChatGPT generate abstractive summaries that are indistinguishable from real summaries to humans?
• RQ2 Can automated metrics quantify differences between real and ChatGPT-generated summaries?
• RQ3 Can a classifier reliably distinguish between ChatGPT-generated and human-written summaries?

Key findings

• Automated metrics show measurable differences between real and ChatGPT-generated summaries (ROUGE-1: 0.30, ROUGE-2: 0.11, ROUGE-L: 0.20, ROUGELSUM: 0.21, METEOR: 0.35).
• Blinded human reviewers failed to distinguish generated from real summaries with accuracy around 0.49.
• Fine-tuned DistillBERT detects ChatGPT-generated vs. real summaries with 90% accuracy (F1 0.33).
• Sentence Embeddings with XGBoost achieved 50% accuracy (baseline).
• The study suggests humans perceive ChatGPT summaries as comparable to real ones when prompts are carefully chosen.