[Paper Review] Advancing Radiograph Representation Learning with Masked Record Modeling

Hong-Yu Zhou, Chenyu Lian+2 more|arXiv (Cornell University)|Jan 30, 2023

COVID-19 diagnosis using AI18 citations

TL;DR

MRM introduces a unified pre-training framework that jointly masks and reconstructs radiograph patches and associated report tokens, yielding strong label-efficient transfer across chest X-ray tasks.

ABSTRACT

Modern studies in radiograph representation learning rely on either self-supervision to encode invariant semantics or associated radiology reports to incorporate medical expertise, while the complementarity between them is barely noticed. To explore this, we formulate the self- and report-completion as two complementary objectives and present a unified framework based on masked record modeling (MRM). In practice, MRM reconstructs masked image patches and masked report tokens following a multi-task scheme to learn knowledge-enhanced semantic representations. With MRM pre-training, we obtain pre-trained models that can be well transferred to various radiography tasks. Specifically, we find that MRM offers superior performance in label-efficient fine-tuning. For instance, MRM achieves 88.5% mean AUC on CheXpert using 1% labeled data, outperforming previous R$^2$L methods with 100% labels. On NIH ChestX-ray, MRM outperforms the best performing counterpart by about 3% under small labeling ratios. Besides, MRM surpasses self- and report-supervised pre-training in identifying the pneumonia type and the pneumothorax area, sometimes by large margins.

Motivation & Objective

• Motivate combining self-supervised and report-supervised radiograph learning to leverage complementary signals.
• Propose a unified pre-training framework that jointly completes masked radiographs and masked radiology reports.
• Learn transferable radiograph representations that generalize under low-label regimes.
• Demonstrate improved performance on multiple public chest X-ray datasets through MRM pre-training.

Proposed method

• Mask random masking on low-resolution radiographs and associated reports with high masking ratios.
• Use a radiograph encoder to produce patch embeddings that support both high-resolution patch restoration and report token restoration.
• Add the global radiograph representation to non-masked report token embeddings to form hybrid representations for report restoration (masked language modeling).
• Restore high-resolution radiograph patches from low-resolution inputs via a patch-based image decoder (MIM loss).
• Train with a multi-task objective combining masked language modeling loss for reports and MSE loss for image restoration (L = L_R + lambda L_I).
• Demonstrate transfer by fine-tuning pre-trained image encoders on downstream classification and segmentation tasks across NIH ChestX-ray, CheXpert, RSNA Pneumonia, SIIM, and COVID-19 datasets.

Experimental results

Research questions

• RQ1 Can self-supervised (image-only) and report-supervised (text-based) radiograph representations be effectively unified?
• RQ2 Does jointly learning masked image reconstruction and masked report restoration improve transfer with limited labels?
• RQ3 How does MRM perform under varying labeling ratios across multiple chest X-ray benchmarks?
• RQ4 What is the contribution of multi-task masking and super-resolution restoration to downstream performance?

Key findings

• MRM achieves 88.5% mean AUC on CheXpert with 1% labeled data, outperforming previous methods with 100% labels.
• On NIH ChestX-ray, MRM outperforms the best counterpart by about 3% under small labeling ratios.
• MRM surpasses self- and report-supervised pre-training in identifying pneumonia type and pneumothorax area in several tasks.
• MRM consistently outperforms CONVIRT, GLoRIA, REFERS and other baselines across CheXpert, RSNA Pneumonia, and SIIM under limited supervision.
• Ablations show that both masked report modeling and masked radiograph modeling contribute to gains, with super-resolution restoration providing additional benefits at various label regimes.