[Paper Review] RADIFUSION: A multi-radiomics deep learning based breast cancer risk prediction model using sequential mammographic images with image attention and bilateral asymmetry refinement

Hong Hui Yeoh, Andrea Liew+6 more|arXiv (Cornell University)|Apr 1, 2023

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

RADIFUSION is a deep learning model that enhances breast cancer risk prediction by integrating sequential mammograms with image attention, radiomic features, a gating mechanism for multi-view fusion, and bilateral asymmetry-based fine-tuning. On the CSAW dataset, it achieved AUCs of 0.905, 0.872, and 0.866 for 1-year, 2-year, and >2-year risk prediction, respectively, outperforming state-of-the-art models.

ABSTRACT

Breast cancer is a significant public health concern and early detection is critical for triaging high risk patients. Sequential screening mammograms can provide important spatiotemporal information about changes in breast tissue over time. In this study, we propose a deep learning architecture called RADIFUSION that utilizes sequential mammograms and incorporates a linear image attention mechanism, radiomic features, a new gating mechanism to combine different mammographic views, and bilateral asymmetry-based finetuning for breast cancer risk assessment. We evaluate our model on a screening dataset called Cohort of Screen-Aged Women (CSAW) dataset. Based on results obtained on the independent testing set consisting of 1,749 women, our approach achieved superior performance compared to other state-of-the-art models with area under the receiver operating characteristic curves (AUCs) of 0.905, 0.872 and 0.866 in the three respective metrics of 1-year AUC, 2-year AUC and > 2-year AUC. Our study highlights the importance of incorporating various deep learning mechanisms, such as image attention, radiomic features, gating mechanism, and bilateral asymmetry-based fine-tuning, to improve the accuracy of breast cancer risk assessment. We also demonstrate that our model's performance was enhanced by leveraging spatiotemporal information from sequential mammograms. Our findings suggest that RADIFUSION can provide clinicians with a powerful tool for breast cancer risk assessment.

Motivation & Objective

• To improve breast cancer risk prediction by leveraging spatiotemporal changes in sequential screening mammograms.
• To integrate radiomic features with deep learning for enhanced feature representation.
• To develop a novel gating mechanism for optimal fusion of different mammographic views (mediolateral oblique and cranio-caudal).
• To refine model performance using bilateral asymmetry analysis as a supervisory signal.
• To demonstrate superior predictive performance compared to existing state-of-the-art models on a real-world screening cohort.

Proposed method

• The model employs a multi-radiomics deep learning architecture that processes sequential mammographic images from two standard views: mediolateral oblique (MLO) and cranio-caudal (CC).
• An image attention mechanism is applied to highlight clinically relevant regions in mammograms, improving feature learning and interpretability.
• Radiomic features extracted from segmented breast regions are concatenated with deep features to enrich representation learning.
• A learnable gating mechanism dynamically combines features from MLO and CC views, optimizing view-specific contributions based on predictive relevance.
• Bilateral asymmetry-based fine-tuning is introduced by training on paired left-right mammographic differences, enhancing sensitivity to subtle asymmetries linked to early cancer.
• The model is trained and validated on the Cohort of Screen-Aged Women (CSAW) dataset, with performance evaluated using time-to-event AUC metrics.

Experimental results

Research questions

• RQ1 Can integrating sequential mammographic images improve long-term breast cancer risk prediction accuracy compared to single-timepoint models?
• RQ2 How does the inclusion of radiomic features and image attention enhance the model's ability to detect early malignant changes?
• RQ3 To what extent does a view-specific gating mechanism improve performance when fusing information from MLO and CC mammographic views?
• RQ4 Does bilateral asymmetry-based fine-tuning significantly improve predictive performance by capturing subtle tissue asymmetries indicative of early disease?
• RQ5 How does RADIFUSION compare to state-of-the-art models in terms of AUC across multiple time horizons (1-year, 2-year, >2-year)?

Key findings

• RADIFUSION achieved an AUC of 0.905 for 1-year breast cancer risk prediction, significantly outperforming existing models.
• The model recorded AUCs of 0.872 and 0.866 for 2-year and >2-year risk prediction, respectively, demonstrating consistent performance across time horizons.
• The integration of image attention improved model focus on suspicious regions, contributing to enhanced discriminative ability.
• The bilateral asymmetry-based fine-tuning strategy significantly boosted model performance by leveraging subtle asymmetries not visible in individual images.
• The gating mechanism effectively balanced contributions from MLO and CC views, with the model learning to prioritize more informative views per patient.
• The model's performance was robust on an independent test set of 1,749 women, confirming its generalizability and clinical potential.