[Paper Review] Multi-Modality Information Fusion for Radiomics-based Neural Architecture Search

Yige Peng, Lei Bi+3 more|arXiv (Cornell University)|Jul 12, 2020

Radiomics and Machine Learning in Medical Imaging22 references4 citations

TL;DR

This paper proposes Multi-Modality Neural Architecture Search (MM-NAS), an automated method that jointly searches for optimal convolutional neural network architectures and fusion strategies in PET-CT radiomics. By enabling end-to-end, architecture-agnostic fusion of PET and CT images, MM-NAS achieves a 3D AUC of 0.896 in predicting distant metastases in soft-tissue sarcoma patients, outperforming state-of-the-art methods without handcrafted features or manual design.

ABSTRACT

'Radiomics' is a method that extracts mineable quantitative features from radiographic images. These features can then be used to determine prognosis, for example, predicting the development of distant metastases (DM). Existing radiomics methods, however, require complex manual effort including the design of hand-crafted radiomic features and their extraction and selection. Recent radiomics methods, based on convolutional neural networks (CNNs), also require manual input in network architecture design and hyper-parameter tuning. Radiomic complexity is further compounded when there are multiple imaging modalities, for example, combined positron emission tomography - computed tomography (PET-CT) where there is functional information from PET and complementary anatomical localization information from computed tomography (CT). Existing multi-modality radiomics methods manually fuse the data that are extracted separately. Reliance on manual fusion often results in sub-optimal fusion because they are dependent on an 'expert's' understanding of medical images. In this study, we propose a multi-modality neural architecture search method (MM-NAS) to automatically derive optimal multi-modality image features for radiomics and thus negate the dependence on a manual process. We evaluated our MM-NAS on the ability to predict DM using a public PET-CT dataset of patients with soft-tissue sarcomas (STSs). Our results show that our MM-NAS had a higher prediction accuracy when compared to state-of-the-art radiomics methods.

Motivation & Objective

• To eliminate manual effort in designing radiomics pipelines, including feature engineering and network architecture selection.
• To address sub-optimal multi-modality fusion in existing radiomics methods, which rely on expert-defined fusion strategies.
• To develop an end-to-end, automated method for learning optimal multi-modality image representations from PET-CT scans.
• To improve prediction accuracy of distant metastases in soft-tissue sarcoma patients using deep learning with minimal human bias.

Proposed method

• Employed differentiable neural architecture search (NAS) to automatically discover optimal computational cells (normal and reduction) for 2D and 3D CNNs.
• Designed a search space that includes various operations (e.g., convolution, pooling) and connections for fusing PET and CT features at multiple stages of the network.
• Integrated multi-modality fusion within the architecture search process, allowing learned fusion modules to emerge naturally during training.
• Used a one-shot weight sharing strategy to efficiently search architectures across 5-fold cross-validation on a public PET-CT STS dataset.
• Trained and evaluated the final architecture using standard deep learning training protocols with early stopping and learning rate scheduling.
• Evaluated performance using 6 metrics: accuracy, precision, F1 score, AUC, sensitivity, and specificity.

Experimental results

Research questions

• RQ1 Can an automated neural architecture search framework outperform manual or handcrafted radiomics pipelines in predicting distant metastases from PET-CT scans?
• RQ2 Does end-to-end joint search for network architecture and fusion strategy lead to better performance than sequential feature extraction and manual fusion?
• RQ3 How does the performance of MM-NAS compare to state-of-the-art radiomics methods that use handcrafted features or fixed architectures?
• RQ4 What is the contribution of 3D volumetric modeling versus 2D slice-based modeling in predicting metastatic potential in soft-tissue sarcomas?
• RQ5 Can learned multi-modality fusion at multiple network stages improve predictive performance compared to late fusion of pre-extracted features?

Key findings

• The 3D MM-NAS achieved the highest AUC of 0.896, outperforming all baseline methods including 3DMCL, HC+RF, and DLHN.
• MM-NAS achieved an accuracy of 0.896, sensitivity of 0.917, specificity of 0.875, precision of 0.880, and F1 score of 0.898 on the 48-patient PET-CT dataset.
• The 3D MM-NAS significantly outperformed 2D and 3D single-modality CNNs (e.g., 3D PET-CT CNN AUC = 0.773), demonstrating the value of joint architecture and fusion search.
• MM-NAS surpassed 3DMCL, a state-of-the-art method combining handcrafted and deep features, without requiring any manual feature engineering.
• The method demonstrated that learned fusion within the network architecture leads to better representation learning than post-hoc fusion of separately extracted features.
• PET-based models outperformed CT-based models, highlighting the functional sensitivity of PET to tumor biology, which is enhanced by end-to-end learning in MM-NAS.