[Paper Review] CheXphoto: 10,000+ Photos and Transformations of Chest X-rays for Benchmarking Deep Learning Robustness

Nick A. Phillips, Pranav Rajpurkar+11 more|arXiv (Cornell University)|Jul 13, 2020

COVID-19 diagnosis using AI14 references17 citations

TL;DR

CheXphoto introduces a large-scale benchmark dataset of 10,000+ smartphone photos and synthetic transformations of chest x-rays, derived from the CheXpert dataset, to evaluate and improve deep learning models' robustness to real-world photographic artifacts. The dataset enables training and testing of models on images resembling those captured in clinical settings using smartphones, significantly enhancing generalization for global deployment in low-resource healthcare environments.

ABSTRACT

Clinical deployment of deep learning algorithms for chest x-ray interpretation requires a solution that can integrate into the vast spectrum of clinical workflows across the world. An appealing approach to scaled deployment is to leverage the ubiquity of smartphones by capturing photos of x-rays to share with clinicians using messaging services like WhatsApp. However, the application of chest x-ray algorithms to photos of chest x-rays requires reliable classification in the presence of artifacts not typically encountered in digital x-rays used to train machine learning models. We introduce CheXphoto, a dataset of smartphone photos and synthetic photographic transformations of chest x-rays sampled from the CheXpert dataset. To generate CheXphoto we (1) automatically and manually captured photos of digital x-rays under different settings, and (2) generated synthetic transformations of digital x-rays targeted to make them look like photos of digital x-rays and x-ray films. We release this dataset as a resource for testing and improving the robustness of deep learning algorithms for automated chest x-ray interpretation on smartphone photos of chest x-rays.

Motivation & Objective

• To address the gap in robustness of deep learning models for chest x-ray interpretation when applied to smartphone-captured images, which differ significantly from digital x-rays in appearance due to photographic artifacts.
• To enable scalable deployment of automated chest x-ray interpretation in global clinical workflows, particularly in regions where x-ray films and smartphones are the norm.
• To develop a publicly available benchmark dataset that captures realistic photographic distortions such as glare, blur, lighting variation, and perspective distortion.
• To support the training and evaluation of models that generalize effectively to real-world imaging conditions encountered in telemedicine and mobile health applications.
• To reduce the performance drop seen in deep learning models when transitioning from digital x-rays to photos of x-rays by including photographic data in model development.

Proposed method

• The dataset was constructed by capturing natural photos of digital chest x-rays displayed on monitors using smartphones (OnePlus 6 and iPhone 8) under varied lighting and positioning conditions, following protocols similar to the iPhone1k dataset.
• Synthetic transformations were generated using a pipeline that applies realistic photographic distortions—such as blur, rotation, translation, glare, and moiré—to digital x-rays to simulate smartphone capture effects.
• The training set includes 10,507 x-rays from 3,000 unique patients sampled from the CheXpert training set, with both natural photos and synthetic transformations.
• The validation set comprises 234 x-rays from 200 patients in the CheXpert validation set, including both natural and synthetic images, plus an additional 200 photos of physical x-ray films from MIMIC-CXR and VinBrain.
• The test set includes 668 x-rays from 500 patients in the CheXpert test set, with photos captured using an iPhone 8, and is withheld for official model evaluation.
• All data is organized by transformation type, with image paths and labels stored in CSV files, and the synthetic generation code is open-sourced for reproducibility.

Experimental results

Research questions

• RQ1 How do deep learning models trained on digital x-rays perform when tested on smartphone-captured images of x-rays, and what is the magnitude of performance degradation?
• RQ2 To what extent can synthetic photographic transformations improve model generalization to real-world smartphone images of x-rays?
• RQ3 Can models trained on a combination of natural photos and synthetic transformations of x-rays achieve robust performance comparable to digital x-ray models on real-world photographic inputs?
• RQ4 How does the inclusion of physical x-ray film photos in the validation set affect model evaluation and generalization to low-resource clinical settings?
• RQ5 What is the impact of diverse photographic artifacts—such as glare, blur, and lighting variation—on the performance of automated chest x-ray classification models?

Key findings

• CheXphoto provides a comprehensive benchmark with over 10,000 images, including natural photos and synthetic transformations of digital x-rays, as well as 200 photos of physical x-ray films, enabling robust evaluation of model generalization.
• The dataset includes 10,507 training images from 3,000 unique patients, 234 validation images from 200 patients in the CheXpert validation set, and 668 test images from 500 patients in the CheXpert test set.
• The validation set contains an additional 200 photos of physical x-ray films from MIMIC-CXR and VinBrain, sourced from real clinical settings in Vietnam and the US.
• The synthetic transformation pipeline successfully replicates common photographic artifacts such as blur, glare, and perspective distortion, enabling data augmentation that mimics real-world smartphone capture.
• The dataset is publicly available with code for synthetic data generation, enabling reproducibility and extension by the research community.
• Model evaluation is conducted via CodaLab, with a private test set and a live leaderboard to ensure fair and reproducible benchmarking of performance on unseen photographic inputs.