[Paper Review] Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports

Qingqing Zhu, Xiuying Chen+7 more|PubMed|Jan 29, 2024

Radiology practices and education7 references5 citations

TL;DR

The paper merges radiologist expertise with in-context instruction learning and chain-of-thought reasoning in LLMs to create a detailed, explainable evaluation framework for AI-generated radiology reports, benchmarked against traditional metrics.

ABSTRACT

In radiology, Artificial Intelligence (AI) has significantly advanced report generation, but automatic evaluation of these AI-produced reports remains challenging. Current metrics, such as Conventional Natural Language Generation (NLG) and Clinical Efficacy (CE), often fall short in capturing the semantic intricacies of clinical contexts or overemphasize clinical details, undermining report clarity. To overcome these issues, our proposed method synergizes the expertise of professional radiologists with Large Language Models (LLMs), like GPT-3.5 and GPT-4. Utilizing In-Context Instruction Learning (ICIL) and Chain of Thought (CoT) reasoning, our approach aligns LLM evaluations with radiologist standards, enabling detailed comparisons between human and AI-generated reports. This is further enhanced by a Regression model that aggregates sentence evaluation scores. Experimental results show that our "Detailed GPT-4 (5-shot)" model achieves a 0.48 score, outperforming the METEOR metric by 0.19, while our "Regressed GPT-4" model shows even greater alignment with expert evaluations, exceeding the best existing metric by a 0.35 margin. Moreover, the robustness of our explanations has been validated through a thorough iterative strategy. We plan to publicly release annotations from radiology experts, setting a new standard for accuracy in future assessments. This underscores the potential of our approach in enhancing the quality assessment of AI-driven medical reports.

Motivation & Objective

• Leverage radiologist expertise to improve automatic evaluation of AI-generated radiology reports.
• Integrate In-Context Instruction Learning (ICIL) and Chain of Thought (CoT) reasoning with large language models (LLMs) for qualitative and quantitative assessment.
• Develop a regression-based aggregation of sentence-level evaluations to produce an overall report score aligned with expert judgments.
• Provide explainable evaluations and plan public release of radiologist annotations to advance accuracy in future assessments.

Proposed method

• Use GPT-3.5 and GPT-4 with ICIL and CoT to evaluate Original vs Predicted radiology reports.
• Radiologists craft instructions and evaluation templates, which guide LLMs to produce per-sentence scores and explanations.
• A regression model aggregates sentence scores into an Overall Score, using features like sentence score ratios (ro0, ro0.5, ro1, ro−1, rp0, rp0.5, rp1, rp−1).
• Iterative Verification: generated explanations and Refined Reports undergo another evaluation loop to validate explanation quality.
• Compare against NLG metrics (BLEU, METEOR, ROUGE-L) and CE metrics (CheXpert labels) and perform human assessment on 100 original-predicted pairs.
• Evaluate alignment with human judgments using Kendall’s Tau and Cohen’s Kappa across multiple GPT configurations (GPT-4 vs GPT-3.5; 1-shot vs 5-shot; simple vs detailed instructions).

Experimental results

Research questions

• RQ1 Can radiologist-guided ICIL with CoT enable LLMs to evaluate radiology reports more closely to expert judgments than traditional metrics?
• RQ2 Do regression-aggregated sentence scores correlate more strongly with expert evaluations than individual sentence scores or standard metrics?
• RQ3 Does providing explanations (CoT) improve the alignment of LLM evaluations with human experts, and how does this vary between GPT-4 and GPT-3.5?
• RQ4 Is the proposed framework robust across different templates, instruction detail levels, and numbers of templates (1-shot vs 5-shot)?
• RQ5 How do radiologist-authored instructions and templates influence the explainability and reliability of AI-assisted evaluations?

Key findings

• The Detailed GPT-4 (5-shot) model achieved higher alignment with expert evaluations than METEOR by 0.19.
• The Regressed GPT-4 model aligned with expert evaluations more strongly, exceeding the best existing metric by a margin of 0.35.
• GPT-4 generally outperformed GPT-3.5 across conditions, with detailed instructions enhancing GPT-4 performance.
• Explanations provided by the LLMs improved agreement with human judgments; without explanations, correlations dropped significantly.
• The Iterative Verification step improved the quality of regenerated reports across all metrics.
• In sentence-level evaluation, 5-shot detailed GPT-4 achieved a Cohen’s Kappa close to radiologist Rater2/Rater3 levels, demonstrating strong expert-like performance.