[Paper Review] MAIRA-2: Grounded Radiology Report Generation

Shruthi Bannur, Kenza Bouzid+19 more|arXiv (Cornell University)|Jun 6, 2024

Radiomics and Machine Learning in Medical Imaging13 citations

TL;DR

MAIRA-2 introduces grounded radiology report generation by pairing a radiology-specific image encoder with an LLM to produce sentence-level grounded findings on chest X-rays, evaluated with RadFact, achieving state-of-the-art findings generation on MIMIC-CXR and enabling grounded reporting.

ABSTRACT

Radiology reporting is a complex task requiring detailed medical image understanding and precise language generation, for which generative multimodal models offer a promising solution. However, to impact clinical practice, models must achieve a high level of both verifiable performance and utility. We augment the utility of automated report generation by incorporating localisation of individual findings on the image - a task we call grounded report generation - and enhance performance by incorporating realistic reporting context as inputs. We design a novel evaluation framework (RadFact) leveraging the logical inference capabilities of large language models (LLMs) to quantify report correctness and completeness at the level of individual sentences, while supporting the new task of grounded reporting. We develop MAIRA-2, a large radiology-specific multimodal model designed to generate chest X-ray reports with and without grounding. MAIRA-2 achieves state of the art on existing report generation benchmarks and establishes the novel task of grounded report generation.

Motivation & Objective

• Motivate automated, high-quality radiology report drafting to alleviate radiologist workload and improve consistency.
• Extend report generation to grounded reporting by localising each finding within the image.
• Develop an evaluation framework (RadFact) that assesses factuality and grounding of generated sentences and their spatial annotations.
• Combine a radiology-specific image encoder with a large language model to handle comprehensive multimodal inputs (current frontal and lateral views, prior studies, Indication, Technique, Comparison).

Proposed method

• Use a Ret- DINO-based radiology image encoder (Rad-DINO) frozen during training to produce image tokens.
• Employ a Vicuna-based LLM (7B or 13B) with a trainable adapter to generate text conditioned on image tokens and structured prompts.
• Represent spatial annotations as discretised 100x100 grid coordinates for bounding boxes, embedded into the generation prompt.
• Provide enriched inputs: current frontal and lateral CXRs, prior frontal image and report, and report sections (Indication, Technique, Comparison) to improve generation fidelity.
• Train in a single stage with autoregressive cross-entropy loss on a multitask dataset mix, with context-length expansion via RoPE scaling to handle up to 3 view images and more inputs.

Experimental results

Research questions

• RQ1 Can MAIRA-2 generate grounded radiology reports that associate findings with precise image locations?
• RQ2 Does including lateral views, prior studies, and detailed report sections improve report quality and grounding accuracy?
• RQ3 How effective is RadFact as an evaluation framework for sentence-level factuality and spatial grounding in radiology reports?
• RQ4 What is the impact of model scale (7B vs 13B) on text quality and grounding performance?

Key findings

• MAIRA-2 achieves grounding-enabled report generation, establishing new state-of-the-art on findings generation (without grounding) on MIMIC-CXR.
• RadFact scores indicate high factuality with logical precision around 73–75% and logical recall around 72–74% for 7B and 13B models.
• Grounding precision is around 68–69% with a high grounding recall (~92–93%), and box-completion precision is ~68–70% with ~84–86% box completion recall.
• 13B variant shows modest text quality gains and more substantial gains in localisation/grounding metrics compared to 7B.
• Box-level grounding (phrase grounding) shows ~68–70% precision and ~84–86% recall for correctly generated phrases.
• RadFact-Llama3 provides a robust, entailment-based evaluation of generated sentences relative to ground-truth references.