[Paper Review] Classifying Symmetrical Differences and Temporal Change in Mammography Using Deep Neural Networks

Thijs Kooi, Nico Karssemeijer|arXiv (Cornell University)|Mar 22, 2017

AI in cancer detection3 citations

TL;DR

This paper proposes a deep learning framework that enhances mammographic mass detection by integrating symmetry and temporal context using two-stream convolutional neural networks (CNNs). It compares two fusion strategies—early fusion of contra-lateral and primary images and a stage-wise feature extraction approach—achieving an AUC of 0.895 for symmetry analysis and 0.884 for temporal change, with statistically significant improvement over baseline in symmetry detection.

ABSTRACT

We investigate the addition of symmetry and temporal context information to a deep Convolutional Neural Network (CNN) with the purpose of detecting malignant soft tissue lesions in mammography. We employ a simple linear mapping that takes the location of a mass candidate and maps it to either the contra-lateral or prior mammogram and Regions Of Interest (ROI) are extracted around each location. We subsequently explore two different architectures (1) a fusion model employing two datastreams were both ROIs are fed to the network during training and testing and (2) a stage-wise approach where a single ROI CNN is trained on the primary image and subsequently used as feature extractor for both primary and symmetrical or prior ROIs. A 'shallow' Gradient Boosted Tree (GBT) classifier is then trained on the concatenation of these features and used to classify the joint representation. Results shown a significant increase in performance using the first architecture and symmetry information, but only marginal gains in performance using temporal data and the other setting. We feel results are promising and can greatly be improved when more temporal data becomes available.

Motivation & Objective

• To improve computer-aided detection (CAD) of malignant masses in mammography by incorporating symmetry and temporal context.
• To investigate whether deep neural networks can effectively learn from contra-lateral breast images and prior mammograms to improve lesion detection.
• To evaluate two distinct fusion architectures for combining symmetry and temporal information in a deep learning framework.
• To determine whether the integration of multi-view and longitudinal data leads to measurable improvements in diagnostic performance over baseline models.
• To explore the feasibility of using deep learning to model asymmetry and change as indicators of malignancy in screening mammography.

Proposed method

• Extract regions of interest (ROIs) around mass candidates from both the primary and contra-lateral mammogram or prior study.
• Employ two distinct CNN architectures: (1) a two-stream fusion model processing both ROIs simultaneously, and (2) a stage-wise model using a pre-trained CNN to extract features from primary and contra-lateral/prior ROIs.
• Concatenate the features from both streams and train a shallow gradient boosted tree (GBT) classifier on the joint representation.
• Use a linear mapping to align mass locations across contra-lateral and prior images to enable consistent ROI extraction.
• Train all networks from scratch, with ablation on fusion strategy and architecture choice.
• Apply confidence intervals and statistical testing (p-value) to assess significance of performance gains over baseline.

Experimental results

Research questions

• RQ1 Can deep neural networks effectively leverage symmetry between left and right breasts to improve detection of malignant masses in mammography?
• RQ2 Does incorporating temporal change from prior mammograms enhance the performance of CAD systems in detecting malignant masses?
• RQ3 Which fusion strategy—early fusion of dual-stream inputs or stage-wise feature extraction—yields better diagnostic performance for symmetry and temporal analysis?
• RQ4 Are the performance gains from symmetry and temporal context statistically significant, particularly at high specificity?
• RQ5 How does the availability of more temporal data potentially affect the scalability and performance of such deep learning models?

Key findings

• The two-stream fusion model achieved an AUC of 0.895 for symmetry analysis, significantly outperforming the baseline AUC of 0.87 (p = 0.02 at high specificity).
• The stage-wise approach achieved an AUC of 0.88 for symmetry, also showing improvement over the baseline, though less pronounced than the fusion model.
• For temporal change detection, the two-stream model achieved an AUC of 0.884, and the stage-wise model reached 0.879, both showing consistent but non-significant improvements over baseline.
• The results suggest that symmetry-based features provide a more substantial performance gain than temporal features, likely due to richer structural cues in bilateral comparisons.
• The authors observe that performance gains from temporal analysis are promising but marginal, and anticipate significant improvement with larger datasets.
• The study demonstrates that deep learning models can effectively integrate multi-modal contextual information (symmetry and time) into CAD systems, with potential for broader application in medical imaging.