[Paper Review] MRScore: Evaluating Radiology Report Generation with LLM-based Reward System

Yunyi Liu, Zhanyu Wang+5 more|arXiv (Cornell University)|Apr 27, 2024

Radiomics and Machine Learning in Medical ImagingMedicine3 citations

TL;DR

This paper proposes MRScore, a novel LLM-based reward model for evaluating radiology report generation that achieves superior alignment with human judgments. By leveraging GPT-4 to generate synthetic, human-like evaluation data across seven clinical and linguistic criteria, and fine-tuning a Mistral-7B-based reward model, MRScore demonstrates significantly higher correlation with radiologist scores than traditional metrics like BLEU, BERTScore, or RadCliQ.

ABSTRACT

In recent years, automated radiology report generation has experienced significant growth. This paper introduces MRScore, an automatic evaluation metric tailored for radiology report generation by leveraging Large Language Models (LLMs). Conventional NLG (natural language generation) metrics like BLEU are inadequate for accurately assessing the generated radiology reports, as systematically demonstrated by our observations within this paper. To address this challenge, we collaborated with radiologists to develop a framework that guides LLMs for radiology report evaluation, ensuring alignment with human analysis. Our framework includes two key components: i) utilizing GPT to generate large amounts of training data, i.e., reports with different qualities, and ii) pairing GPT-generated reports as accepted and rejected samples and training LLMs to produce MRScore as the model reward. Our experiments demonstrate MRScore's higher correlation with human judgments and superior performance in model selection compared to traditional metrics. Our code and datasets will be available on GitHub.

Motivation & Objective

• To address the poor alignment of conventional NLG metrics like BLEU with human radiologist evaluations in radiology report generation.
• To overcome the high cost and limited scalability of human-annotated training data for report evaluation models.
• To develop a framework that encodes radiologist expertise into a structured, interpretable evaluation system for automated report scoring.
• To train a reward model using LLM-generated synthetic evaluation data to achieve human-like scoring performance without requiring extensive human-annotated labels.
• To improve the reliability and interpretability of automated evaluation by incorporating clinical accuracy, linguistic quality, and structural completeness.

Proposed method

• Design a seven-criteria evaluation framework based on radiologist input, covering clinical findings, linguistic quality, structure, and consistency.
• Use GPT-4 to generate 3,000 synthetic report pairs (accepted/rejected) for 1,000 reference reports, each labeled with quality scores across the seven criteria.
• Train a reward model on the paired reports using the Mistral-7B-instruct backbone, with the reward output being the MRScore.
• Fine-tune the reward model via supervised learning on the synthetic data, where the model learns to predict a scalar score reflecting report quality.
• Evaluate the model’s performance using Kendall’s Tau and Spearman’s correlation against human radiologist ratings.
• Compare MRScore’s performance against baseline metrics including BLEU, ROUGE, METEOR, CIDEr, BERTScore, RadGraph F1, and RadCliQ.

Experimental results

Research questions

• RQ1 Can GPT-4 reliably generate synthetic radiology report evaluations that align with human radiologist judgments?
• RQ2 Does an LLM-based reward model trained on synthetic expert-like evaluations achieve higher correlation with human evaluations than traditional NLG metrics?
• RQ3 How does MRScore compare to hybrid metrics like RadCliQ in terms of correlation with human ratings and robustness to linguistic variation?
• RQ4 To what extent does incorporating clinical and linguistic criteria into a multi-dimensional evaluation framework improve model selection and report quality assessment?
• RQ5 Can a reward model trained on synthetic data achieve performance comparable to models trained on human-annotated data, especially in low-resource settings?

Key findings

• MRScore achieved a Kendall’s Tau of 0.250 and a Spearman’s rho of 0.304 with human radiologist ratings, significantly outperforming traditional metrics like BLEU (p-value 0.688) and ROUGE (p-value 0.429).
• MRScore demonstrated the highest correlation with human evaluations among all tested metrics, with a Spearman’s coefficient of 0.304, surpassing BERTScore (0.200) and RadGraph F1 (0.176).
• The Mistral-7B-based reward model achieved the best performance among backbone models, with a Kendall’s Tau of 0.250 and a p-value of 0.002, indicating strong statistical significance.
• MRScore showed a statistically significant correlation with human ratings (p-value = 0.002), while traditional NLG metrics like BLEU and ROUGE showed no significant correlation (p > 0.05).
• The framework’s reliance on GPT-4 for synthetic data generation enabled large-scale, low-cost training without requiring human-annotated labels, while maintaining high alignment with expert judgment.
• RadCliQ-based metrics showed moderate correlation (Spearman’s rho = -0.241), but MRScore’s positive and stronger correlation indicates better alignment with human evaluation trends.