[Paper Review] Deep Learning in Medical Image Registration: Magic or Mirage?

Rohit Jena, Deeksha Sethi+2 more|arXiv (Cornell University)|Aug 11, 2024

Radiomics and Machine Learning in Medical ImagingMedicine3 citations

TL;DR

This paper investigates whether deep learning (DLIR) methods truly outperform classical optimization-based registration in medical imaging. It finds that classical methods generalize better across datasets due to strong correlation between per-pixel intensity-label mutual information and performance, while DLIR methods, despite high fidelity on training data, fail to generalize under domain shift—making classical methods more reliable unless large labeled datasets are available for retraining.

ABSTRACT

Classical optimization and learning-based methods are the two reigning paradigms in deformable image registration. While optimization-based methods boast generalizability across modalities and robust performance, learning-based methods promise peak performance, incorporating weak supervision and amortized optimization. However, the exact conditions for either paradigm to perform well over the other are shrouded and not explicitly outlined in the existing literature. In this paper, we make an explicit correspondence between the mutual information of the distribution of per-pixel intensity and labels, and the performance of classical registration methods. This strong correlation hints to the fact that architectural designs in learning-based methods is unlikely to affect this correlation, and therefore, the performance of learning-based methods. This hypothesis is thoroughly validated with state-of-the-art classical and learning-based methods. However, learning-based methods with weak supervision can perform high-fidelity intensity and label registration, which is not possible with classical methods. Next, we show that this high-fidelity feature learning does not translate to invariance to domain shift, and learning-based methods are sensitive to such changes in the data distribution. Finally, we propose a general recipe to choose the best paradigm for a given registration problem, based on these observations.

Motivation & Objective

• To resolve the ongoing debate on whether deep learning-based or classical optimization-based methods are superior for deformable medical image registration.
• To identify the conditions under which each paradigm excels, particularly regarding generalization and robustness to domain shift.
• To investigate whether label matching in deep learning improves performance transfer across different neuroimaging datasets.
• To establish a principled framework for selecting the optimal registration paradigm based on data availability and distribution characteristics.

Proposed method

• The authors establish a strong correlation between per-pixel intensity and label map mutual information and the performance of classical registration methods, using this as a key diagnostic for method suitability.
• They evaluate state-of-the-art classical and deep learning-based registration methods (e.g., ANTs, FireANTs, SynthMorph, TransMorph) across four T1-weighted brain imaging datasets with different labeling protocols and acquisition parameters.
• Performance is measured using mean Dice score over labeled anatomical regions, with models trained on OASIS and tested on CUMC12, LPBA40, MGH10, and IBSR18 datasets to assess domain shift robustness.
• The study compares supervised and unsupervised DLIR variants, particularly assessing the impact of label matching loss during training on generalization performance.
• A systematic ablation is performed to isolate the effects of architectural design and loss function in DLIR, showing that performance is primarily constrained by mutual information rather than network architecture.
• A general recipe for selecting between classical and deep learning methods is proposed based on data distribution, labeling availability, and required robustness.

Experimental results

Research questions

• RQ1 What is the relationship between per-pixel intensity-label mutual information and classical registration performance?
• RQ2 Does incorporating label maps during training improve generalization of deep learning-based registration methods across different datasets?
• RQ3 How do classical and deep learning-based methods compare in performance and robustness under domain shift across multiple neuroimaging datasets?
• RQ4 Under what conditions does deep learning-based registration outperform classical optimization-based methods in medical image registration?

Key findings

• Classical registration methods (e.g., ANTs, FireANTs) consistently outperform deep learning-based methods across four independent T1-brain datasets, even when the latter are trained with label matching.
• The performance of classical methods correlates strongly with the mutual information between per-pixel intensity and label maps, suggesting this metric is a key predictor of registration success.
• Despite high performance on the OASIS dataset, DLIR methods like SynthMorph and TransMorph fail to generalize to other datasets, with no significant improvement from label matching loss.
• DLIR methods trained on one dataset do not achieve better performance on other datasets, even with access to their label maps, indicating poor domain transfer.
• Classical methods maintain state-of-the-art performance across diverse datasets without retraining, due to their agnosticism to modality, resolution, and preprocessing protocols.
• The study concludes that classical methods are more reliable for general use, while DLIR should only be used when large, domain-specific labeled datasets are available for retraining.