[Paper Review] A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization

Griffin Adams, Jason Zucker+1 more|PubMed|Mar 7, 2023

Topic Modeling88 references10 citations

TL;DR

The paper benchmarks a broad set of faithfulness metrics against fine-grained clinician annotations for long-form hospital-course summaries, examining domain adaptation, source–summary alignment, and metric distillation; finds shorter, sentence-level inputs yield the strongest human correlations.

ABSTRACT

Long-form clinical summarization of hospital admissions has real-world significance because of its potential to help both clinicians and patients. The factual consistency of summaries-their faithfulness-is critical to their safe usage in clinical settings. To better understand the limitations of state-of-the-art natural language processing (NLP) systems, as well as the suitability of existing evaluation metrics, we benchmark faithfulness metrics against fine-grained human annotations for model-generated summaries of a patient's Brief Hospital Course. We create a corpus of patient hospital admissions and summaries for a cohort of HIV patients, each with complex medical histories. Annotators are presented with summaries and source notes, and asked to categorize manually highlighted summary elements (clinical entities like conditions and medications as well as actions like "following up") into one of three categories: "Incorrect," "Missing," and "Not in Notes." We meta-evaluate a broad set of faithfulness metrics-proposed for the general NLP domain-by measuring the correlation of metric scores to clinician ratings. Across metrics, we explore the importance of domain adaptation (e.g. the impact of in-domain pre-training and metric fine-tuning), the use of source-summary alignments, and the effects of distilling a single metric from an ensemble. We find that off-the-shelf metrics with no exposure to clinical text correlate well to clinician ratings yet overly rely on copy-and-pasted text. As a practical guide, we observe that most metrics correlate best to clinicians when provided with one summary sentence at a time and a minimal set of supporting sentences from the notes before discharge.

Motivation & Objective

• Collect fine-grained, sentence- and element-level faithfulness annotations for long-form hospital-course summaries.
• Benchmark a broad set of faithfulness metrics against clinician judgments on a HIV patient cohort.
• Investigate how domain adaptation, input length, and source–summary alignments affect metric performance.
• Explore combining metrics via ensemble methods and distilling a single superior faithfulness metric.

Proposed method

• Fine-tune a Longformer Encoder-Decoder (LED) on a large hospital-course corpus to generate long-form summaries.
• Collect expert annotations on a held-out HIV-cohort set to label fidelity of summary elements relative to source notes.
• Benchmark multiple faithfulness metrics (e.g., BARTScore, BERTScore, SummaC, CTC) under various domain adaptation, input-length, and alignment settings.
• Implement three domain adaptation levels for each metric: Off-The-Shelf (out-of-domain), Tuned In-Domain, and Double In-Domain.
• Evaluate multiple source–summary alignment strategies (sentence-level, section-level, entity-chain, and full-input) and their impact on metric performance.
• Distill a metric from an ensemble of baseline metrics to improve correlation with human judgments.

Experimental results

Research questions

• RQ1 What source input granularity yields the strongest correlation with human faithfulness judgments for long-form clinical summaries?
• RQ2 How does domain adaptation (in-domain pre-training and tuning) impact metric performance on clinical long-form summaries?
• RQ3 What is the effect of different source–summary alignment strategies on metric reliability and correlation to human judgments?
• RQ4 Should metric tuning align with the usage scenario (alignment method during tuning vs inference) for optimal performance?
• RQ5 Can distillation from an ensemble of metrics produce a superior single faithfulness metric?

Key findings

• Off-the-shelf metrics correlate with human judgments but tend to overemphasize extractiveness.
• Metrics generally correlate best with human judgments when evaluated sentence-by-sentence rather than on full summaries.
• Shorter, more relevant source alignments yield higher and more stable correlations than using the full source input.
• In-domain adaptation provides limited gains in raw correlation, but gains emerge when focusing on abstractive subsets of annotations.
• Distilling a metric from an ensemble of baseline metrics improves correlation with expert labels over individual metrics.
• Entity-based and alignment-aware approaches (e.g., top-section, entity-chain alignments) offer competitive performance compared to broader strategies.