[Paper Review] GPTScore: Evaluate as You Desire

Jinlan Fu, See-Kiong Ng+2 more|arXiv (Cornell University)|Feb 8, 2023

Topic Modeling81 citations

TL;DR

GPTScore uses large pre-trained language models with zero-shot instructions and in-context examples to score generated text across multiple customizable evaluation aspects, without annotated data.

ABSTRACT

Generative Artificial Intelligence (AI) has enabled the development of sophisticated models that are capable of producing high-caliber text, images, and other outputs through the utilization of large pre-trained models. Nevertheless, assessing the quality of the generation is an even more arduous task than the generation itself, and this issue has not been given adequate consideration recently. This paper proposes a novel evaluation framework, GPTScore, which utilizes the emergent abilities (e.g., zero-shot instruction) of generative pre-trained models to score generated texts. There are 19 pre-trained models explored in this paper, ranging in size from 80M (e.g., FLAN-T5-small) to 175B (e.g., GPT3). Experimental results on four text generation tasks, 22 evaluation aspects, and corresponding 37 datasets demonstrate that this approach can effectively allow us to achieve what one desires to evaluate for texts simply by natural language instructions. This nature helps us overcome several long-standing challenges in text evaluation--how to achieve customized, multi-faceted evaluation without the need for annotated samples. We make our code publicly available at https://github.com/jinlanfu/GPTScore.

Motivation & Objective

• Address the lack of flexible, multi-aspect text evaluation that minimizes annotation and training requirements.
• Leverage emergent abilities of large generative Models (zero-shot instruction, in-context learning) for scoring generated text.
• Provide a protocol to define task, aspects, and context to compute conditional generation probabilities as scores.
• Compare multiple backbones (GPT-2, OPT, FLAN-T5, GPT-3) across diverse NLG tasks to assess reliability and generality.

Proposed method

• Propose GPTScore, which computes GPTScore(h|d,a,S) as a weighted sum of log probabilities of tokens in h conditioned on the prompt T(d,a,S) and context S (p(y|x,θ)).
• Define evaluation protocol via task description d and aspect definition a, with optional exemplar samples to guide the model via in-context learning.
• Use a prompt template T(·) that encodes the evaluation protocol and aspect definitions for different tasks and aspects.
• Instantiate GPTScore with backbones including GPT-3, OPT, FLAN-T5, and GPT-2 to explore decoder-only and encoder-decoder architectures.
• Explore two scoring variants: src->hypo (based on source) and ref->hypo (based on gold reference) to align with human judgments across tasks.

Experimental results

Research questions

• RQ1 Can instruction-based prompting and in-context demonstrations enable training-free, multi-aspect evaluation of generated text across diverse NLP tasks?
• RQ2 How do different backbones (GPT-3, GPT-2, OPT, FLAN-T5, GPT-J) compare in reliability and correlation with human judgments across datasets?
• RQ3 What is the impact of using source versus reference as the scoring basis for various tasks and aspects?
• RQ4 To what extent do task definitions and aspect definitions influence the reliability of GPTScore across domains (Summarization, Dialogue, Data-to-Text, MT)?

Key findings

• Instruction-based prompting significantly improves GPTScore performance across many settings.
• Decoder-only models often benefit more from instruction and demonstrations than encoder-decoder models.
• GPT3-d03 generally outperforms GPT3-d01, but GPT3-c01 offers a cost-efficient alternative with competitive results in MT.
• Across tasks, GPTScore with instruction can outperform some supervised or traditional metrics in correlation with human judgments.
• GPTScore demonstrates consistent gains when combining instruction with demonstrations (IDM) for multiple model families.
• Across summarization, MT, data-to-text, and dialogue tasks, GPTScore variants show meaningful correlations with human judgments (as reported in the experiments).