[Paper Review] PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation

Haibo Jin, Haoxuan Che+2 more|arXiv (Cornell University)|Aug 24, 2023

Topic ModelingComputer Science3 citations

TL;DR

PromptMRG proposes a diagnosis-driven prompt framework for medical report generation that improves diagnostic accuracy by integrating a disease classification branch with token-based prompts to guide text generation. It achieves state-of-the-art clinical efficacy performance on two benchmarks by combining cross-modal feature enhancement and self-adaptive disease-balanced learning to address class imbalance.

ABSTRACT

Automatic medical report generation (MRG) is of great research value as it has the potential to relieve radiologists from the heavy burden of report writing. Despite recent advancements, accurate MRG remains challenging due to the need for precise clinical understanding and disease identification. Moreover, the imbalanced distribution of diseases makes the challenge even more pronounced, as rare diseases are underrepresented in training data, making their diagnostic performance unreliable. To address these challenges, we propose diagnosis-driven prompts for medical report generation (PromptMRG), a novel framework that aims to improve the diagnostic accuracy of MRG with the guidance of diagnosis-aware prompts. Specifically, PromptMRG is based on encoder-decoder architecture with an extra disease classification branch. When generating reports, the diagnostic results from the classification branch are converted into token prompts to explicitly guide the generation process. To further improve the diagnostic accuracy, we design cross-modal feature enhancement, which retrieves similar reports from the database to assist the diagnosis of a query image by leveraging the knowledge from a pre-trained CLIP. Moreover, the disease imbalanced issue is addressed by applying an adaptive logit-adjusted loss to the classification branch based on the individual learning status of each disease, which overcomes the barrier of text decoder's inability to manipulate disease distributions. Experiments on two MRG benchmarks show the effectiveness of the proposed method, where it obtains state-of-the-art clinical efficacy performance on both datasets. The code is available at https://github.com/jhb86253817/PromptMRG.

Motivation & Objective

• To address the poor diagnostic accuracy in existing medical report generation (MRG) models, which often fail to generate clinically correct reports despite strong linguistic fluency.
• To improve the performance on rare diseases, which suffer from data imbalance and low diagnostic reliability in current MRG systems.
• To bridge the gap between MRG models and clinical needs by explicitly guiding report generation using diagnostic predictions from a dedicated classification branch.
• To overcome the text decoder’s inability to control disease distribution by introducing a self-adaptive loss mechanism for class-imbalanced learning.

Proposed method

• A dual-branch encoder-decoder architecture is used, with an additional disease classification branch to generate diagnostic predictions for each input image.
• Diagnosis-driven prompts (DDP) convert classification results into token prompts that are injected into the decoder to guide report generation with explicit diagnostic context.
• Cross-modal feature enhancement (CFE) leverages a pre-trained CLIP model to retrieve similar image-report pairs from a database, enhancing the query image’s features via dynamic aggregation.
• Self-adaptive disease-balanced learning (SDL) applies an adaptive logit-adjusted loss to the classification branch, adjusting optimization objectives based on each disease’s individual learning status.
• The framework integrates these components end-to-end, enabling the decoder to generate reports that are both linguistically fluent and diagnostically accurate.

Experimental results

Research questions

• RQ1 Can diagnosis-driven prompts improve the clinical efficacy of medical report generation models?
• RQ2 How does cross-modal feature enhancement using CLIP improve disease classification and report generation accuracy?
• RQ3 Can self-adaptive loss training effectively mitigate the performance drop on rare diseases in imbalanced MRG datasets?
• RQ4 Does integrating diagnostic classification with report generation lead to better alignment with ground-truth clinical findings than standard MRG methods?

Key findings

• PromptMRG achieves state-of-the-art clinical efficacy (CE) performance on both MIMIC-CXR and NIH ChestX-ray datasets, outperforming existing SOTA methods.
• The use of token prompts leads to a 6.4% absolute improvement in F1 score over the baseline, significantly outperforming text and feature prompts.
• Rare diseases saw an average F1 improvement of 8% with the SDL module, with some diseases improving by over 12%.
• The F1 score for the rarest diseases increased substantially, though the average F1 score only rose by 1.2%, indicating that standard metrics may underweight rare disease performance.
• The CFE module enhanced diagnostic accuracy by retrieving semantically and visually similar reports using CLIP, improving feature representation for classification.
• The method demonstrates that explicit diagnostic prompting is essential, as feature and embedding prompts performed worse than token prompts, highlighting the importance of structured diagnostic input.