[Paper Review] Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction

Matthew J. Muckley, Bruno Riemenschneider+21 more|arXiv (Cornell University)|Dec 9, 2020

Radiomics and Machine Learning in Medical ImagingMedicine29 citations

TL;DR

This paper presents the results of the 2020 fastMRI Challenge, a machine learning competition focused on accelerating MRI reconstruction using subsampled k-space data from 7,299 de-identified brain scans. The challenge introduced a new Transfer track to evaluate model generalization across vendors and emphasized radiologist assessment of pathology depiction, resulting in a state-of-the-art model that outperformed others in both SSIM scores and clinical evaluation.

ABSTRACT

Accelerating MRI scans is one of the principal outstanding problems in the MRI research community. Towards this goal, we hosted the second fastMRI competition targeted towards reconstructing MR images with subsampled k-space data. We provided participants with data from 7,299 clinical brain scans (de-identified via a HIPAA-compliant procedure by NYU Langone Health), holding back the fully-sampled data from 894 of these scans for challenge evaluation purposes. In contrast to the 2019 challenge, we focused our radiologist evaluations on pathological assessment in brain images. We also debuted a new Transfer track that required participants to submit models evaluated on MRI scanners from outside the training set. We received 19 submissions from eight different groups. Results showed one team scoring best in both SSIM scores and qualitative radiologist evaluations. We also performed analysis on alternative metrics to mitigate the effects of background noise and collected feedback from the participants to inform future challenges. Lastly, we identify common failure modes across the submissions, highlighting areas of need for future research in the MRI reconstruction community.

Motivation & Objective

• Address the clinical need for faster MRI scans by accelerating image reconstruction using machine learning.
• Improve model generalization across different MRI scanner vendors and sites, a key barrier to clinical deployment.
• Enhance evaluation relevance by shifting from overall image quality to clinical pathology depiction in radiologist assessments.
• Provide a large-scale, publicly available dataset with multi-coil, subsampled k-space data for reproducible research.
• Identify limitations in current metrics and model behavior, such as hallucinations and noise sensitivity, to guide future method development.

Proposed method

• Retrospectively undersampled fully-sampled k-space data from 7,299 clinical brain MRI scans using pseudo-equispaced sampling masks to achieve exact 4X and 8X acceleration rates.
• Used a strong baseline model based on an End-to-End Variational Network to set a high performance threshold for participants.
• Introduced a new 'Transfer' track requiring models to be evaluated on data from MRI scanners not present in the training set, testing cross-vendor generalization.
• Maintained the fully-sampled center of k-space to support autocalibration for parallel imaging and compressed sensing methods.
• Evaluated submissions quantitatively using SSIM and qualitatively via radiologist readings focused on pathological structure visibility.
• Collected participant feedback to inform future challenge design, particularly regarding compute demands and realism of sampling patterns.

Experimental results

Research questions

• RQ1 Can machine learning models trained on multi-coil, single-vendor data generalize effectively to unseen MRI scanner vendors?
• RQ2 How does focusing radiologist evaluation on pathology depiction rather than overall image quality affect model ranking and clinical relevance?
• RQ3 To what extent do current metrics like SSIM and RSS fail to capture clinically relevant image quality, especially in the presence of noise or artifacts?
• RQ4 What are the common failure modes in deep learning-based MRI reconstruction models, particularly at high acceleration factors?
• RQ5 How can evaluation protocols be improved to reduce the risk of hallucinations and better align with clinical image interpretation?

Key findings

• One team achieved the best performance in both SSIM scores and radiologist evaluations, setting a new state-of-the-art for MRI reconstruction under 8X acceleration.
• The Transfer track revealed significant performance drops on out-of-distribution scanners, highlighting the challenge of cross-vendor generalization in clinical deployment.
• Radiologists showed mixed sentiment toward images from the 8X and Transfer tracks, indicating that pathology depiction remains a difficult and open research frontier.
• Participants reported that the pseudo-equispaced sampling masks used in the challenge did not fully replicate real vendor sequences, suggesting models may need further fine-tuning for clinical use.
• There was widespread concern among participants about the high computational cost of training state-of-the-art models, with some using only a single GPU, indicating a barrier to entry for academic researchers.
• Feedback indicated that SSIM favors smoothing and may conflict with radiologist preferences for diagnostic detail, suggesting a need for improved metrics that better align with human perception and clinical needs.