[Paper Review] DeViDe: Faceted medical knowledge for improved medical vision-language pre-training

Haozhe Luo, Ziyu Zhou+3 more|arXiv (Cornell University)|Apr 4, 2024

Medical and Biological Sciences4 citations

TL;DR

DeViDe proposes a novel vision-language pre-training framework for chest X-rays that integrates multi-faceted medical knowledge—radiology reports, medical definitions, and open-web radiographic visual descriptions—using a bi-directional transformer with multi-granularity alignment losses. It achieves state-of-the-art zero-shot performance on three large-scale datasets and superior fine-tuning results across classification and segmentation tasks, with visual descriptions contributing up to 6.6% AUC gain in some pathologies.

ABSTRACT

Vision-language pre-training for chest X-rays has made significant strides, primarily by utilizing paired radiographs and radiology reports. However, existing approaches often face challenges in encoding medical knowledge effectively. While radiology reports provide insights into the current disease manifestation, medical definitions (as used by contemporary methods) tend to be overly abstract, creating a gap in knowledge. To address this, we propose DeViDe, a novel transformer-based method that leverages radiographic descriptions from the open web. These descriptions outline general visual characteristics of diseases in radiographs, and when combined with abstract definitions and radiology reports, provide a holistic snapshot of knowledge. DeViDe incorporates three key features for knowledge-augmented vision language alignment: First, a large-language model-based augmentation is employed to homogenise medical knowledge from diverse sources. Second, this knowledge is aligned with image information at various levels of granularity. Third, a novel projection layer is proposed to handle the complexity of aligning each image with multiple descriptions arising in a multi-label setting. In zero-shot settings, DeViDe performs comparably to fully supervised models on external datasets and achieves state-of-the-art results on three large-scale datasets. Additionally, fine-tuning DeViDe on four downstream tasks and six segmentation tasks showcases its superior performance across data from diverse distributions.

Motivation & Objective

• To address the gap in effective medical knowledge integration in vision-language pre-training for chest X-rays, particularly the over-reliance on raw radiology reports and abstract definitions.
• To improve model alignment by incorporating detailed, multi-source medical knowledge at varying levels of granularity.
• To develop a knowledge-augmented VLP framework that enhances zero-shot and few-shot generalization across diverse clinical data distributions.
• To investigate the impact of radiographic visual descriptions on model performance and interpretability in medical imaging.
• To create a unified, scalable method for aligning image features with diverse textual medical knowledge without requiring additional annotations.

Proposed method

• DeViDe employs a bi-directional transformer architecture that processes image features, radiology reports, medical definitions, and open-web-derived visual descriptions of radiological entities in a unified input format.
• It uses a large language model to homogenize and standardize medical knowledge from heterogeneous sources, ensuring consistency in terminology and structure.
• The model is trained with two novel loss functions: a patch-to-text alignment loss ($\mathcal{L}_{pta}$) for local image-text correspondence and a text-to-image contrastive loss ($\mathcal{L}_{tnc}$) for global alignment.
• A novel projection layer enables multi-label image-to-description alignment, allowing a single image to be associated with multiple detailed visual descriptions.
• Knowledge injection is performed via a dedicated knowledge-retrieval layer that uses cross-attention to ground textual descriptions on relevant image regions.
• The framework operates without additional annotations, relying solely on weakly supervised signals from paired images and reports.

Experimental results

Research questions

• RQ1 How does incorporating radiographic visual descriptions from the open web improve zero-shot performance in medical vision-language pre-training?
• RQ2 To what extent do multi-granularity losses ($\mathcal{L}_{pta}$ and $\mathcal{L}_{tnc}$) enhance alignment between image regions and textual knowledge?
• RQ3 Can DeViDe achieve state-of-the-art performance in zero-shot settings without fine-tuning, compared to fully supervised baselines?
• RQ4 How does DeViDe’s initialization improve data efficiency and convergence in few-shot and fine-tuning scenarios?
• RQ5 What is the contribution of detailed visual descriptions to model interpretability and localization accuracy in abnormality detection?

Key findings

• DeViDe achieves state-of-the-art zero-shot performance on three large-scale datasets: ChestX-ray14 (AUC 0.7771), CheXpert (AUC 0.8998), and PadChest (AUC 0.7465).
• In zero-shot evaluation, DeViDe matches the performance of fully supervised models on external datasets, demonstrating strong generalization.
• The ablation study shows that removing $\mathcal{L}_{pta}$ reduces AUC by up to 3% on two datasets, confirming its critical role in local alignment.
• The injection of visual descriptions leads to performance gains of up to 6.6% in AUC for the Fracture class on PadChest, highlighting their impact.
• When fine-tuned on four classification and six segmentation tasks, DeViDe-initialized models converge to better optima and show improved data efficiency.
• Qualitative analysis confirms that attention maps align well with anatomical regions, with diffuse findings like 'diffuse nodule' receiving broader attention and localized findings like 'nodule' receiving focused attention.