[Paper Review] A Critical Evaluation of Evaluations for Long-form Question Answering

Fangyuan Xu, Yixiao Song+2 more|arXiv (Cornell University)|May 29, 2023

Topic Modeling4 citations

TL;DR

This paper presents the first comprehensive evaluation of long-form question answering (LFQA) by comparing human expert judgments with automatic metrics. It finds that no existing automatic metric reliably predicts human preference, but some correlate with fine-grained aspects like coherence and factuality, advocating for multi-faceted evaluation over single overall scores. The authors release expert-annotated data and code to advance future LFQA evaluation research.

ABSTRACT

Long-form question answering (LFQA) enables answering a wide range of questions, but its flexibility poses enormous challenges for evaluation. We perform the first targeted study of the evaluation of long-form answers, covering both human and automatic evaluation practices. We hire domain experts in seven areas to provide preference judgments over pairs of answers, along with free-form justifications for their choices. We present a careful analysis of experts' evaluation, which focuses on new aspects such as the comprehensiveness of the answer. Next, we examine automatic text generation metrics, finding that no existing metrics are predictive of human preference judgments. However, some metrics correlate with fine-grained aspects of answers (e.g., coherence). We encourage future work to move away from a single "overall score" of the answer and adopt a multi-faceted evaluation, targeting aspects such as factuality and completeness. We publicly release all of our annotations and code to spur future work into LFQA evaluation.

Motivation & Objective

• To investigate the reliability and validity of current evaluation practices for long-form question answering (LFQA).
• To assess whether existing automatic text generation metrics correlate with human preference judgments for LFQA answers.
• To identify key quality dimensions—such as factuality, completeness, and coherence—that experts prioritize over surface-level features.
• To advocate for a shift from single-score human evaluations to multi-faceted evaluation frameworks targeting specific answer qualities.
• To release a new expert-annotated dataset of 260 preference judgments with justifications to support future LFQA evaluation research.

Proposed method

• Hired domain experts in seven fields (e.g., biology, economics) to perform pairwise preference judgments on LFQA answers, providing detailed justifications.
• Collected 140 answer pairs across 260 expert ratings, with annotations focusing on factuality, completeness, coherence, and other fine-grained aspects.
• Evaluated 12 existing automatic metrics (e.g., ROUGE, QAFactEval, RankGen) for correlation with human judgments on overall quality and specific aspects.
• Trained a learned metric (LERC) using a pipeline combining named entity recognition, question generation, answer verification, and similarity scoring via fine-tuned BART and Electra models.
• Pilot-tested GPT-3 text-davinci-003 for few-shot pair-wise evaluation using in-context learning with two-shot examples and temperature sampling.
• Used a multi-stage evaluation pipeline: (1) extract answers from sources, (2) generate questions, (3) verify answers, (4) compute similarity scores using learned metrics.

Experimental results

Research questions

• RQ1 How do expert judgments on long-form answers differ from crowdworker judgments in terms of evaluation criteria?
• RQ2 Which aspects of answer quality—such as factuality, completeness, or coherence—are most decisive in expert preference judgments?
• RQ3 To what extent do existing automatic metrics correlate with human preference judgments for LFQA answers?
• RQ4 Can learned metrics be trained to predict human preference judgments effectively, or do they only capture partial aspects of quality?
• RQ5 What are the limitations of using a single overall score to evaluate LFQA answers, and what alternatives are more effective?

Key findings

• Experts prioritize factuality and completeness over surface-level features like conciseness, and often disagree due to differing valuations of fine-grained aspects.
• No existing automatic metric reliably correlates with human preference judgments for overall answer quality, indicating a major gap in current evaluation practices.
• Metrics like QAFactEval and RankGen show moderate correlation with fine-grained aspects such as faithfulness and coherence, suggesting potential for targeted metric development.
• The study reveals that human evaluators, even experts, exhibit significant disagreement, highlighting the complexity of defining answer quality in LFQA.
• The authors release a new dataset of 260 expert-annotated answer pairs with justifications, which can serve as a benchmark for future LFQA evaluation research.
• Pilot results show GPT-3 can achieve high consistency and moderate alignment with human preferences in few-shot settings, though further investigation is needed.