[Paper Review] Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images

Xiaoman Zhang, Chaoyi Wu+3 more|arXiv (Cornell University)|Feb 27, 2023

COVID-19 diagnosis using AI4 citations

TL;DR

This paper proposes Knowledge-enhanced Auto Diagnosis (KAD), a vision-language pre-training model for chest X-ray analysis that integrates medical knowledge from a knowledge graph to improve zero-shot and few-shot diagnostic performance. By leveraging clinical entities and relations extracted from radiology reports and a query-based transformer (Disease Query Network), KAD achieves zero-shot performance comparable to fully supervised models and surpasses the average of three expert radiologists on three out of five pathologies with statistical significance.

ABSTRACT

While multi-modal foundation models pre-trained on large-scale data have been successful in natural language understanding and vision recognition, their use in medical domains is still limited due to the fine-grained nature of medical tasks and the high demand for domain knowledge. To address this challenge, we propose a novel approach called Knowledge-enhanced Auto Diagnosis (KAD) which leverages existing medical domain knowledge to guide vision-language pre-training using paired chest X-rays and radiology reports. We evaluate KAD on {four} external X-ray datasets and demonstrate that its zero-shot performance is not only comparable to that of fully-supervised models, but also superior to the average of three expert radiologists for three (out of five) pathologies with statistical significance. Moreover, when few-shot annotation is available, KAD outperforms all existing approaches in fine-tuning settings, demonstrating its potential for application in different clinical scenarios.

Motivation & Objective

• To address the limitations of existing vision-language models in medical imaging, which struggle with fine-grained diagnosis and lack domain-specific knowledge.
• To improve zero-shot and few-shot generalization in chest X-ray diagnosis by incorporating structured medical knowledge from a knowledge graph.
• To enhance model interpretability by generating attention maps that localize clinically relevant regions in X-ray images.
• To reduce reliance on large-scale annotated data by leveraging weakly supervised pre-training on paired images and reports.
• To demonstrate that knowledge-enhanced pre-training can match or exceed human expert performance on unseen pathologies without fine-tuning.

Proposed method

• A two-stage training framework: first, pre-train a knowledge encoder on a medical knowledge graph using contrastive learning to learn dense textual representations of medical entities and relations.
• Extract clinical entities and relations from radiology reports using three methods: heuristic rules, RadGraph, or GPT-3.5 (ChatGPT), to generate structured knowledge inputs.
• Perform image-text contrastive learning between chest X-ray features and the embeddings of extracted medical entities and relations, aligning visual and knowledge-augmented textual representations.
• Train a Disease Query Network (DQN), a query-based transformer that takes a disease name as input and attends to visual features to produce classification probabilities and attention maps.
• Use the pre-trained knowledge encoder and DQN for zero-shot inference by encoding the disease name as a query and generating attention maps that highlight relevant image regions.
• Enable few-shot fine-tuning using standard protocols, demonstrating strong transferability when additional annotations are available.

Experimental results

Research questions

• RQ1 Can knowledge-enhanced vision-language pre-training improve zero-shot diagnostic performance on chest X-ray images compared to existing self-supervised and supervised models?
• RQ2 Does integrating a medical knowledge graph into visual-language pre-training lead to better generalization on rare or unseen pathologies?
• RQ3 Can the model’s predictions be made interpretable through attention maps that align with radiologists’ annotations?
• RQ4 How does the model’s performance compare to that of expert radiologists in zero-shot evaluation across multiple pathologies?
• RQ5 To what extent does knowledge injection improve data efficiency in few-shot fine-tuning scenarios?

Key findings

• KAD achieves zero-shot performance on PadChest that is significantly higher than state-of-the-art self-supervised models for 193 out of 193 pathologies, with statistical significance.
• On CheXpert, KAD outperforms the average of three expert radiologists on three out of five pathologies in zero-shot evaluation, with statistical significance.
• In few-shot fine-tuning, KAD surpasses all existing approaches, demonstrating superior transferability and data efficiency.
• The model generates attention maps that closely align with radiologists’ lesion annotations, as confirmed by pointing game evaluation on ChestX-Det10.
• KAD provides reliable, grounded visual explanations through cross-attention maps, enhancing clinical trust and interpretability.
• The integration of knowledge graph representations improves model generalization, especially for long-tailed and rare pathologies.