[Paper Review] Effect of Radiology Report Labeler Quality on Deep Learning Models for Chest X-Ray Interpretation

Saahil Jain, Akshay Smit+2 more|arXiv (Cornell University)|Apr 1, 2021

COVID-19 diagnosis using AI4 citations

TL;DR

This study evaluates the impact of radiology report labeler quality on deep learning models for chest X-ray interpretation, demonstrating that using the more advanced VisualCheXbert labeler significantly improves both label extraction accuracy and downstream image classification performance. The key finding is that models trained on labels from VisualCheXbert achieve a weighted average AUROC of 0.87, substantially outperforming models trained on labels from CheXpert and CheXbert, which yield 0.83 and 0.79 respectively.

ABSTRACT

Although deep learning models for chest X-ray interpretation are commonly trained on labels generated by automatic radiology report labelers, the impact of improvements in report labeling on the performance of chest X-ray classification models has not been systematically investigated. We first compare the CheXpert, CheXbert, and VisualCheXbert labelers on the task of extracting accurate chest X-ray image labels from radiology reports, reporting that the VisualCheXbert labeler outperforms the CheXpert and CheXbert labelers. Next, after training image classification models using labels generated from the different radiology report labelers on one of the largest datasets of chest X-rays, we show that an image classification model trained on labels from the VisualCheXbert labeler outperforms image classification models trained on labels from the CheXpert and CheXbert labelers. Our work suggests that recent improvements in radiology report labeling can translate to the development of higher performing chest X-ray classification models.

Motivation & Objective

• To investigate how improvements in radiology report labeler quality affect the performance of deep learning models for chest X-ray classification.
• To compare the accuracy of label extraction from radiology reports using three state-of-the-art labelers: CheXpert, CheXbert, and VisualCheXbert.
• To evaluate whether higher-quality labels from advanced labelers translate into better-performing image classification models on the CheXpert dataset.
• To assess the impact of uncertainty handling strategies (U-Zeros and U-Ones) on labeler performance and downstream model accuracy.
• To establish a causal link between report labeler quality and the performance of end-to-end chest X-ray classification systems.

Proposed method

• The study uses the CheXpert dataset, which includes 224,316 chest X-ray images with expert-annotated labels for 8 conditions.
• Three radiology report labelers—CheXpert, CheXbert, and VisualCheXbert—are evaluated for their ability to extract accurate image labels from radiology reports.
• Uncertain predictions from CheXpert and CheXbert are handled using two strategies: mapping to negative (U-Zeros) or positive (U-Ones) class.
• Image classification models are trained using a DenseNet-121 architecture on labels generated by each labeler, with training conducted on 3 TITAN-XP GPUs.
• Model performance is evaluated on the CheXpert test set using AUROC scores with 95% confidence intervals computed via non-parametric percentile bootstrap (1000 replicates).
• Statistical significance of performance differences is assessed using bootstrapped confidence intervals to compare labeler and model performance across conditions.

Experimental results

Research questions

• RQ1 Does the quality of radiology report labelers significantly affect the accuracy of extracted image labels for chest X-rays?
• RQ2 How do uncertainty handling strategies (U-Zeros vs. U-Ones) influence the performance of automatic labelers?
• RQ3 Can improvements in radiology report labeling translate into better-performing deep learning models for chest X-ray classification?
• RQ4 Is there a measurable performance gap between models trained on labels from CheXpert, CheXbert, and VisualCheXbert?
• RQ5 Does the use of a more advanced labeler like VisualCheXbert lead to statistically significant improvements in downstream image classification AUROC?

Key findings

• The VisualCheXbert labeler achieves a weighted average F1 score of 0.73 (95% CI: 0.71–0.75), significantly outperforming CheXpert (0.50) and CheXbert (0.50) under U-Zeros, and CheXpert (0.55) and CheXbert (0.55) under U-Ones.
• The image classification model trained on labels from VisualCheXbert achieves a weighted average AUROC of 0.87 (95% CI: 0.86–0.89), which is statistically superior to models trained on labels from CheXpert (0.83) and CheXbert (0.79).
• For the condition 'Enlarged Cardiom.' (n=253), VisualCheXbert achieved an F1 of 0.73, compared to 0.23 for CheXbert under U-Ones, highlighting its robustness on rare conditions.
• The model trained on VisualCheXbert labels achieved an AUROC of 0.84 for Atelectasis, compared to 0.70 for CheXpert (U-Zeros), indicating improved detection of subtle or complex findings.
• On the condition 'Lung Opacity' (n=264), the VisualCheXbert-based model achieved an AUROC of 0.91, outperforming CheXpert (0.90) and CheXbert (0.89), showing consistent gains across high-prevalence conditions.
• The study confirms that recent advancements in radiology report labeling directly translate into measurable improvements in deep learning model performance for chest X-ray interpretation.