[Paper Review] CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT

Akshay Smit, Saahil Jain+4 more|arXiv (Cornell University)|Apr 20, 2020

Radiomics and Machine Learning in Medical ImagingMedicine74 citations

TL;DR

CheXbert fine-tunes a biomedically pretrained BERT on rule-based labeler outputs and small expert annotations augmented with backtranslation, achieving state-of-the-art radiology report labeling on MIMIC-CXR test data and nearing radiologist performance.

ABSTRACT

The extraction of labels from radiology text reports enables large-scale training of medical imaging models. Existing approaches to report labeling typically rely either on sophisticated feature engineering based on medical domain knowledge or manual annotations by experts. In this work, we introduce a BERT-based approach to medical image report labeling that exploits both the scale of available rule-based systems and the quality of expert annotations. We demonstrate superior performance of a biomedically pretrained BERT model first trained on annotations of a rule-based labeler and then finetuned on a small set of expert annotations augmented with automated backtranslation. We find that our final model, CheXbert, is able to outperform the previous best rules-based labeler with statistical significance, setting a new SOTA for report labeling on one of the largest datasets of chest x-rays.

Motivation & Objective

• Motivate scalable radiology report labeling to enable large-scale medical imaging model training.
• Leverage existing rule-based labelers to bootstrap a biomedical language model.
• Improve labeling accuracy with limited expert annotations augmented via backtranslation.
• Demonstrate that combining rule-based outputs and expert labels yields superior performance over prior labelers.

Proposed method

• Use a BERT-base architecture with 14 task-specific linear heads for 14 observations.
• Initialize from a biomedically pretrained BERT model trained on rule-based labeler outputs, then fine-tune on expert annotations.
• Augment a small expert-labeled corpus with automated backtranslation to diversify expressions.
• Evaluate using a weighted-F1 metric across 14 observations and three retrieval tasks (positive, negative, uncertain).
• Compare to CheXpert baseline and radiologist benchmark across CheXpert and MIMIC-CXR datasets.

Experimental results

Research questions

• RQ1 Can a biomedically pretrained BERT model, initialized from rule-based labeler outputs, outperform pure expert-label training and pure automatic labeling?
• RQ2 Does backtranslation augmentation improve labeling performance for radiology reports?
• RQ3 How close can CheXbert approach radiologist-level labeling performance on large chest X-ray datasets?
• RQ4 What is the relative performance gain when using biomedical-pretrained versus general pretrained representations in this task?

Key findings

• CheXbert (Tblue-hybrid-bt) achieves F1 = 0.798 (95% CI 0.775, 0.816), outperforming CheXpert (0.743) with a significant difference (p<0.001).
• CheXbert is 0.007 F1 points below radiologist benchmark (0.805, 0.784–0.823).
• Backtranslation augmentation improves performance over non-augmented variants (e.g., T-blue-hybrid-bt vs T-blue-rad).
• On per-condition analysis, CheXbert yields largest gains for Pneumonia (0.151), Fracture (0.120), Consolidation (0.105), Enlarged Cardiomediastinum (0.100), and No Finding (0.097).
• CheXbert outperforms models trained only on radiologist labels or only on automatic labeler outputs across most observations.