[Paper Review] Effectively Fine-tune to Improve Large Multimodal Models for Radiology Report Generation

Yuzhe Lu, Sungmin Hong+2 more|arXiv (Cornell University)|Dec 3, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes a two-stage fine-tuning strategy to improve large multimodal models for radiology report generation by aligning visual features to the text embedding space via soft visual prompts. Using OpenLLaMA-7B with this method, the model achieves state-of-the-art performance without domain-specific pretraining, significantly improving factual correctness as measured by F1-CheXBert scores.

ABSTRACT

Writing radiology reports from medical images requires a high level of domain expertise. It is time-consuming even for trained radiologists and can be error-prone for inexperienced radiologists. It would be appealing to automate this task by leveraging generative AI, which has shown drastic progress in vision and language understanding. In particular, Large Language Models (LLM) have demonstrated impressive capabilities recently and continued to set new state-of-the-art performance on almost all natural language tasks. While many have proposed architectures to combine vision models with LLMs for multimodal tasks, few have explored practical fine-tuning strategies. In this work, we proposed a simple yet effective two-stage fine-tuning protocol to align visual features to LLM's text embedding space as soft visual prompts. Our framework with OpenLLaMA-7B achieved state-of-the-art level performance without domain-specific pretraining. Moreover, we provide detailed analyses of soft visual prompts and attention mechanisms, shedding light on future research directions.

Motivation & Objective

• To improve the clinical accuracy of radiology report generation using large language models (LLMs) without domain-specific pretraining.
• To address the challenge of effectively aligning visual features with LLMs' text embedding space for multimodal generation.
• To investigate whether fine-tuning the vision encoder improves performance compared to freezing it, contrary to common practice.
• To analyze attention mechanisms in large models to understand their grounding to visual inputs.
• To provide a practical, scalable framework for leveraging off-the-shelf LLMs in medical report generation.

Proposed method

• A lightweight mapping network, implemented as a single transformer decoder layer, projects visual features from a vision encoder into the text embedding dimension of a causal language model.
• The mapping network performs both attention pooling (reducing 225 visual tokens to 10) and feature projection to match the language model’s embedding size.
• The projected visual features are treated as soft visual prompts and conditioned into the language model’s autoregressive generation process.
• A two-stage fine-tuning protocol is introduced: first warming up the mapping network, then jointly fine-tuning the vision encoder and mapping network while keeping the LLM frozen.
• Attention weights are normalized and averaged across layers and samples to measure how much each generated token attends to the visual prompts.
• The F1-CheXBert metric is used to evaluate factual completeness by comparing model-generated reports to ground truth using a CheX-BERT-based labeler.

Figure 1: We observed that fine-tuning the visual encoder together with the remaining model components consistently improves F1-CXB14 score.

Experimental results

Research questions

• RQ1 Does fine-tuning the vision encoder improve performance in radiology report generation compared to freezing it?
• RQ2 How does the attention allocation to soft visual prompts vary across different-sized language models?
• RQ3 Can a two-stage fine-tuning protocol enhance factual correctness and clinical efficacy in LLM-based report generation?
• RQ4 Why does performance gain from scaling up LLMs (e.g., from GPT2 to OpenLLaMA-7B) plateau despite lower validation loss?
• RQ5 To what extent are large LLMs grounded to visual inputs when generating radiology reports?

Key findings

• The two-stage fine-tuning strategy improved F1-CheXBert-14 scores by an average of 1.9 points across all model sizes.
• The OpenLLaMA-7B model with two-stage fine-tuning outperformed BioVIL-T, despite BioVIL-T having access to twice as much MIMIC-CXR data and domain-specific pretraining.
• OpenLLaMA-7B allocated significantly less attention to soft visual prompts than smaller GPT2 models, suggesting weaker visual grounding in larger models.
• The average pairwise similarity between generated reports increased with OpenLLaMA-7B, indicating that the language modeling loss alone failed to separate visual features by class.
• The model achieved state-of-the-art performance on both NLG and clinical evaluation metrics without any domain-specific pretraining.
• Visualization of attention weights confirmed that larger LLMs like OpenLLaMA-7B are less reliant on visual prompts, highlighting a key challenge in multimodal alignment.

Figure 2: We visualize the average attention weights allocated to the visual prompts across all attention layers. One critical observation is that OpenLLaMA-7B gives much less attention to soft visual prompts than smaller GPT2 models. This suggests that OpenLLaMA-7B, and potentially larger language