[Paper Review] Contrastive Learning of Medical Visual Representations from Paired Images and Text

Yuhao Zhang, Hang Jiang+3 more|arXiv (Cornell University)|Oct 2, 2020

Multimodal Machine Learning ApplicationsComputer Science38 references278 citations

TL;DR

ConVIRT pretrains medical image encoders with bidirectional image-text contrastive learning on paired reports, yielding superior in-domain representations and data efficiency compared to ImageNet and other baselines.

ABSTRACT

Learning visual representations of medical images (e.g., X-rays) is core to medical image understanding but its progress has been held back by the scarcity of human annotations. Existing work commonly relies on fine-tuning weights transferred from ImageNet pretraining, which is suboptimal due to drastically different image characteristics, or rule-based label extraction from the textual report data paired with medical images, which is inaccurate and hard to generalize. Meanwhile, several recent studies show exciting results from unsupervised contrastive learning from natural images, but we find these methods help little on medical images because of their high inter-class similarity. We propose ConVIRT, an alternative unsupervised strategy to learn medical visual representations by exploiting naturally occurring paired descriptive text. Our new method of pretraining medical image encoders with the paired text data via a bidirectional contrastive objective between the two modalities is domain-agnostic, and requires no additional expert input. We test ConVIRT by transferring our pretrained weights to 4 medical image classification tasks and 2 zero-shot retrieval tasks, and show that it leads to image representations that considerably outperform strong baselines in most settings. Notably, in all 4 classification tasks, our method requires only 10\% as much labeled training data as an ImageNet initialized counterpart to achieve better or comparable performance, demonstrating superior data efficiency.

Motivation & Objective

• Motivate learning high-quality medical image representations with limited annotated data in healthcare.
• Leverage naturally paired medical images and descriptive reports to improve visual encoders without extra expert labeling.
• Evaluate transferability of ConVIRT-pretrained encoders to multiple medical imaging tasks and retrieval settings.

Proposed method

• Represent images and texts as d-dimensional vectors through modality-specific encoders and projection heads.
• Use a bidirectional contrastive objective with two asymmetric losses: image-to-text and text-to-image, combined as a weighted sum.
• Sample random image views and text spans to create diverse positive pairs for contrastive learning.
• Pretrain image encoder (ResNet50) and text encoder (BERT-based ClinicalBERT) on paired data from MIMIC-CXR and a musculoskeletal dataset.
• Apply data augmentations suitable for medical images (crop, flip, affine, color jitter, Gaussian blur) and sentence-level text sampling.
• Evaluate pretrained encoders via linear classification and fine-tuning on four medical classification tasks and perform zero-shot image-image and text-image retrieval.

Experimental results

Research questions

• RQ1 Can cross-modality contrastive learning between medical images and their paired descriptive text yield better visual representations than image-only or random initializations?
• RQ2 Does ConVIRT improve data efficiency, achieving competitive performance with substantially less labeled data compared to ImageNet-pretrained models?
• RQ3 How well do ConVIRT representations transfer to diverse medical imaging tasks and to zero-shot retrieval settings?

Key findings

• ConVIRT generally outperforms random, ImageNet, and in-domain baselines across four classification tasks in both linear and fine-tuning settings.
• On three of four tasks, with only 1% labeled data, ConVIRT matches or surpasses ImageNet initialization with 100% data.
• In zero-shot retrieval, ConVIRT achieves the best Precision@k across image-image and text-image tasks.
• Compared to image-only contrastive methods (SimCLR, MoCo v2), ConVIRT provides substantial gains by leveraging paired text.
• Saliency analyses suggest ConVIRT focuses on more relevant anatomical regions than ImageNet- or other baselines.