[Paper Review] MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation

Yen Nhi Truong Vu, Richard Wang+4 more|arXiv (Cornell University)|Feb 21, 2021

COVID-19 diagnosis using AI19 references27 citations

TL;DR

MedAug uses patient metadata to form cross-image positive pairs in contrastive pretraining, boosting chest X-ray pleural effusion classification performance by leveraging multiple images per patient. Negative-pair strategies with metadata showed no clear gains.

ABSTRACT

Self-supervised contrastive learning between pairs of multiple views of the same image has been shown to successfully leverage unlabeled data to produce meaningful visual representations for both natural and medical images. However, there has been limited work on determining how to select pairs for medical images, where availability of patient metadata can be leveraged to improve representations. In this work, we develop a method to select positive pairs coming from views of possibly different images through the use of patient metadata. We compare strategies for selecting positive pairs for chest X-ray interpretation including requiring them to be from the same patient, imaging study or laterality. We evaluate downstream task performance by fine-tuning the linear layer on 1% of the labeled dataset for pleural effusion classification. Our best performing positive pair selection strategy, which involves using images from the same patient from the same study across all lateralities, achieves a performance increase of 14.4% in mean AUC from the ImageNet pretrained baseline. Our controlled experiments show that the keys to improving downstream performance on disease classification are (1) using patient metadata to appropriately create positive pairs from different images with the same underlying pathologies, and (2) maximizing the number of different images used in query pairing. In addition, we explore leveraging patient metadata to select hard negative pairs for contrastive learning, but do not find improvement over baselines that do not use metadata. Our method is broadly applicable to medical image interpretation and allows flexibility for incorporating medical insights in choosing pairs for contrastive learning.

Motivation & Objective

• Motivate improving self-supervised contrastive learning for medical images using patient metadata.
• Develop a flexible method (MedAug) to form positive pairs from multiple images sharing patient properties.
• Quantify downstream gains on chest X-ray pleural effusion classification with limited labels.
• Analyze which metadata (patient, study, laterality) most benefits representation quality.
• Explore negative-pair strategies leveraging metadata and assess their impact.

Proposed method

• Propose MedAug to generate enhanced augmentation sets by collecting images from the same patient that satisfy criteria c (e.g., same study, same/different laterality).
• Use MoCo v2 pretraining with ResNet-18, 20 epochs, batch size 16, learning rate 1e-4, temperature 0.2, and an MLP projection.
• Fine-tune a linear classifier on frozen encoder outputs and also perform end-to-end fine-tuning with 1% labeled data.
• Evaluate different criteria c (same patient, various study/laterality configurations) to form positive pairs.
• Compare against ImageNet and MoCo v2 baselines and report mean AUC with standard deviation.
• Optionally analyze negative-pair strategies based on metadata (laterality) and their effects.

Experimental results

Research questions

• RQ1 Does incorporating patient metadata to form positive pairs improve pretrained representations for medical imaging tasks?
• RQ2 Which metadata cues (patient, study, laterality) most improve downstream pleural effusion classification performance?
• RQ3 Is increasing the number of distinct images in positive pairs beneficial for representation quality?
• RQ4 Do metadata-informed strategies for negative pairs yield gains over standard MoCo baselines?
• RQ5 How do metadata-based positive pairs perform across different CheXpert tasks beyond pleural effusion?

Key findings

• Best positive-pair strategy (same patient, same study, all lateralities) yields a 3.4% linear AUC improvement and 2.4% end-to-end AUC improvement over the baseline MoCo v2 setup.
• Using images from the same patient to select positive pairs generally improves AUC, except when including images with different study numbers in some settings.
• Increasing the pool of images for positive pairs (same patient, same study, all lateralities) improves downstream performance.
• Negative-pair strategies using metadata (especially laterality) did not surpass the default metadata-free MoCo baseline in AUC.
• Across CheXpert tasks, the best metadata-based approach (same patient, same study) consistently outperformed ImageNet and MoCo baselines for linear fine-tuning.
• Exploratory negative-pair analyses did not yield consistent gains.