[Paper Review] RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training

Zheng Yuan, Qiao Jin+5 more|arXiv (Cornell University)|Mar 1, 2023

Multimodal Machine Learning Applications5 citations

TL;DR

This paper proposes RAMM, a retrieval-augmented biomedical visual question answering framework that leverages a new patient-based image-text dataset (PMCPM) and a retrieval-attention mechanism to enhance performance on limited biomedical VQA data. By retrieving semantically similar image-text pairs from PubMed using contrastive representation learning and fusing them via a novel attention module, RAMM achieves state-of-the-art results on Med-VQA2019, Med-VQA2021, VQARAD, and SLAKE, with scores of 82.13, 39.20, 78.27, and 86.05, respectively.

ABSTRACT

Vision-and-language multi-modal pretraining and fine-tuning have shown great success in visual question answering (VQA). Compared to general domain VQA, the performance of biomedical VQA suffers from limited data. In this paper, we propose a retrieval-augmented pretrain-and-finetune paradigm named RAMM for biomedical VQA to overcome the data limitation issue. Specifically, we collect a new biomedical dataset named PMCPM which offers patient-based image-text pairs containing diverse patient situations from PubMed. Then, we pretrain the biomedical multi-modal model to learn visual and textual representation for image-text pairs and align these representations with image-text contrastive objective (ITC). Finally, we propose a retrieval-augmented method to better use the limited data. We propose to retrieve similar image-text pairs based on ITC from pretraining datasets and introduce a novel retrieval-attention module to fuse the representation of the image and the question with the retrieved images and texts. Experiments demonstrate that our retrieval-augmented pretrain-and-finetune paradigm obtains state-of-the-art performance on Med-VQA2019, Med-VQA2021, VQARAD, and SLAKE datasets. Further analysis shows that the proposed RAMM and PMCPM can enhance biomedical VQA performance compared with previous resources and methods. We will open-source our dataset, codes, and pretrained model.

Motivation & Objective

• To address the data scarcity challenge in biomedical visual question answering (VQA), which limits model generalization and increases overfitting risk.
• To develop a retrieval-augmented pretrain-and-finetune paradigm that leverages external biomedical literature to enrich training signals.
• To construct a high-quality, patient-centric image-text dataset (PMCPM) from PubMed to support biomedical multi-modal pretraining and retrieval.
• To design a novel retrieval-attention module that effectively fuses retrieved image-text pairs with input images and questions during fine-tuning.
• To demonstrate that retrieval from diverse, high-quality literature sources improves VQA performance beyond standard fine-tuning on existing datasets.

Proposed method

• Proposes PMCPM, a large-scale, patient-based image-text dataset collected from PubMed Central, containing diverse radiological images and their corresponding clinical descriptions.
• Pretrains a multi-modal model using three objectives: masked language modeling, image-text matching, and image-text contrastive learning (ITC) to align visual and textual representations.
• Uses ITC embeddings from pretraining to retrieve semantically similar image-text pairs from PMCPM, ROCO, and MIMIC-CXR for each input VQA sample.
• Introduces a retrieval-attention module that attends to both the input image and question and the representations of retrieved image-text pairs to generate enriched context representations.
• Fine-tunes the model with the fused representations to predict answers, enabling knowledge transfer from retrieved literature.
• Employs a dynamic retrieval count strategy, with ablation showing optimal performance at 2–4 retrieved pairs, avoiding noise from excessive retrieval.

Figure 1 : Example of a similar image whose text includes helpful information for answering biomedical questions. The left part is a biomedical-related question with a provided image. The right part is a similar image in PubMed with a relevant caption that indicates the answer (text in red) to the q

Experimental results

Research questions

• RQ1 Can leveraging external biomedical literature through retrieval improve performance on low-resource biomedical VQA datasets?
• RQ2 How effective is a newly constructed, patient-focused image-text dataset (PMCPM) compared to existing datasets like ROCO and MIMIC-CXR in supporting biomedical VQA?
• RQ3 Does the integration of retrieved image-text pairs via a retrieval-attention mechanism lead to better answer accuracy than standard fine-tuning?
• RQ4 What is the optimal number of retrieved image-text pairs for boosting VQA performance without introducing noise?
• RQ5 To what extent do retrieved texts contain answers to biomedical VQA questions, and how does this vary across different datasets?

Key findings

• RAMM achieves state-of-the-art performance on Med-VQA2019 with a score of 82.13, surpassing previous methods.
• On Med-VQA2021, RAMM achieves a score of 39.20, demonstrating strong performance on challenging, open-ended biomedical VQA.
• For the VQARAD dataset, RAMM obtains a score of 78.27, indicating effective handling of radiology-specific VQA tasks.
• On the SLAKE dataset, RAMM achieves 86.05, with a notable 91.59 on closed-ended questions, showing strong generalization.
• Ablation studies confirm that both the PMCPM dataset and the retrieval-augmented fine-tuning mechanism significantly improve performance.
• Retrieval from PMCPM contributes to 29.6% to 38.7% of answers being directly found in retrieved texts, validating the utility of the retrieval strategy.

Figure 2 : The workflow of our proposed retrieval-augmented paradigm RAMM for biomedical VQA. The origin image is from VQA datasets, and the other images are retrieved. Blue lines represent retrieving image-text pairs (PMCID:4061445, 2278144, 4725742, and 3616602) from PMCPM, ROCO, and MIMIC-CXR dat