[Paper Review] RadBERT-CL: Factually-Aware Contrastive Learning For Radiology Report Classification

Ajay Jaiswal, Liyan Tang+4 more|PubMed|Oct 28, 2021

Radiology practices and education31 references12 citations

TL;DR

RadBERT-CL introduces a factually-aware contrastive learning framework that enhances radiology report classification by preserving critical medical nuances like negation and uncertainty through novel data augmentation techniques. Pre-trained on weakly labeled MIMIC-CXR data using a contrastive loss, it achieves state-of-the-art performance, outperforming BERT and CheXbert by 6–11% in F1-score when fine-tuned on limited labeled data.

ABSTRACT

Radiology reports are unstructured and contain the imaging findings and corresponding diagnoses transcribed by radiologists which include clinical facts and negated and/or uncertain statements. Extracting pathologic findings and diagnoses from radiology reports is important for quality control, population health, and monitoring of disease progress. Existing works, primarily rely either on rule-based systems or transformer-based pre-trained model fine-tuning, but could not take the factual and uncertain information into consideration, and therefore generate false positive outputs. In this work, we introduce three sedulous augmentation techniques which retain factual and critical information while generating augmentations for contrastive learning. We introduce RadBERT-CL, which fuses these information into BlueBert via a self-supervised contrastive loss. Our experiments on MIMIC-CXR show superior performance of RadBERT-CL on fine-tuning for multi-class, multi-label report classification. We illustrate that when few labeled data are available, RadBERT-CL outperforms conventional SOTA transformers (BERT/BlueBert) by significantly larger margins (6-11%). We also show that the representations learned by RadBERT-CL can capture critical medical information in the latent space.

Motivation & Objective

• To address the limitation of existing models in capturing factual nuances such as negation and uncertainty in radiology reports.
• To improve radiology report classification performance under low-resource settings with few labeled examples.
• To develop data augmentation techniques that preserve clinical facts while enabling effective contrastive pre-training.
• To demonstrate that contrastive learning can learn discriminative, factually aware representations in medical text.

Proposed method

• Proposes three novel data augmentation techniques at sentence and document levels that preserve factual and uncertain information, including negation and uncertainty markers.
• Fine-tunes BlueBert using a self-supervised contrastive loss to align positive report snippets while pushing apart negative ones.
• Integrates an Info-Preservation Module to detect and retain negated and uncertain statements during augmentation.
• Uses weakly labeled MIMIC-CXR data for pre-training, avoiding reliance on expensive expert-annotated data.
• Employs a contrastive learning objective that maximizes agreement between augmented views of the same report while minimizing agreement across different reports.
• Evaluates performance via linear evaluation and fine-tuning on multi-class, multi-label classification tasks.

Experimental results

Research questions

• RQ1 Can contrastive learning with fact-preserving data augmentation improve the representation of radiology reports in the latent space?
• RQ2 Does RadBERT-CL outperform existing SOTA models like BERT and CheXbert in low-resource settings with limited labeled data?
• RQ3 Can the model’s embeddings distinguish between subtle factual differences such as presence vs. absence or certainty vs. uncertainty in radiology reports?
• RQ4 How effective is the proposed contrastive pre-training strategy in capturing medical nuances without requiring manual annotations?

Key findings

• RadBERT-CL outperforms CheXbert by 6–11% in weighted F1-score when fine-tuned on only 400 labeled reports, demonstrating superior performance in low-data regimes.
• The model achieves state-of-the-art performance on 11 out of 14 findings in the MIMIC-CXR dataset, with consistent gains across multiple evaluation metrics.
• Cosine similarity analysis shows RadBERT-CL embeddings can distinguish between factually different report snippets—e.g., 'no consolidation' vs. 'definite consolidation'—while BERT and BlueBert fail to do so.
• Linear evaluation results confirm that RadBERT-CL learns significantly better representations than BERT and BlueBert, even without fine-tuning.
• The Info-Preservation Module successfully retains negated and uncertain statements during augmentation, which contributes to improved factual awareness in the latent space.
• Pre-training on the full MIMIC-CXR dataset before fine-tuning yields better performance than pre-training on a smaller subset, highlighting the benefit of large-scale unsupervised data.