[Paper Review] F3RNet: Full-Resolution Residual Registration Network for Multimodal Image Registration

Zhe Xu, Jie Luo+3 more|arXiv (Cornell University)|Sep 15, 2020

Medical Image Segmentation Techniques17 references4 citations

TL;DR

F3RNet is a novel unsupervised deep learning framework for multimodal CT-MRI registration that combines full-resolution feature learning with multi-scale residual representations through parallel streams, achieving sub-3-second inference time and superior accuracy on severely deformed abdominal organs, especially in hard-to-align local regions.

ABSTRACT

PURPOSE: Multimodal deformable image registration is essential for many image-guided therapies. Recently, deep learning approaches have gained substantial popularity and success in deformable image registration. Most deep learning approaches use the so-called mono-stream “high-to-low, low-to-high” network structure, and can achieve satisfactory overall registration results. However, accurate alignments for some severely deformed local regions, which are crucial for pinpointing surgical targets, are often overlooked, especially for multimodal inputs with vast intensity differences. Consequently, these approaches are not sensitive to some hard-to-align regions, e.g., intra-patient registration of deformed liver lobes. METHODS: We propose a novel unsupervised registration network, namely Full-Resolution Residual Registration Network (F3RNet), for multimodal registration of severely deformed organs. The proposed method combines two parallel processing streams in a residual learning fashion. One stream takes advantage of the full-resolution information that facilitates accurate voxel-level registration. The other stream learns the deep multi-scale residual representations to obtain robust recognition. We also factorize the 3D convolution to reduce the training parameters and enhance network efficiency. RESULTS: We validate the proposed method on 50 sets of clinically acquired intra-patient abdominal CT-MRI data. Experiments on both CT-to-MRI and MRI-to-CT registration demonstrate promising results compared to state-of-the-art approaches. CONCLUSION: By combining the high-resolution information and multi-scale representations in a highly interactive residual learning fashion, the proposed F3RNet can achieve accurate overall and local registration. The run time for registering a pair of CT-MRI images is less than 3 seconds using a GPU. In future works, we will investigate how to cost-effectively process high-resolution information and fuse multi-scale representations.

Motivation & Objective

• To address the challenge of inaccurate local alignment in multimodal deformable image registration, particularly for severely deformed organs like liver lobes.
• To improve sensitivity to hard-to-align regions with large intensity differences between CT and MRI.
• To develop a lightweight, efficient network that maintains high-resolution spatial details while capturing robust multi-scale features.
• To reduce model complexity and training parameters through factorized 3D convolutions without sacrificing performance.

Proposed method

• F3RNet employs a dual-stream architecture: one stream preserves full-resolution features for precise voxel-level alignment, while the other learns deep multi-scale residual representations for robust feature understanding.
• The two streams are fused via residual learning, enabling interactive feature refinement and improved representation learning.
• 3D convolutions are factorized into depthwise separable convolutions to reduce parameter count and enhance training efficiency.
• The network is trained in an unsupervised manner using a similarity loss between the deformed moving image and the fixed image, without requiring ground-truth deformation fields.

Experimental results

Research questions

• RQ1 Can a dual-stream network architecture combining full-resolution and multi-scale features improve local registration accuracy in multimodal CT-MRI images?
• RQ2 How does residual learning between high-resolution and multi-scale streams enhance alignment in severely deformed regions?
• RQ3 To what extent does factorized 3D convolution reduce model complexity while preserving registration performance?
• RQ4 Can the proposed method achieve real-time inference (under 3 seconds per pair) on clinical 3D abdominal scans?

Key findings

• F3RNet achieved state-of-the-art performance on 50 sets of clinical intra-patient abdominal CT-MRI data for both CT-to-MRI and MRI-to-CT registration.
• The method demonstrated superior accuracy in local regions with severe deformation, particularly in challenging areas like liver lobes.
• Registration of a single CT-MRI pair was completed in less than 3 seconds using a GPU, indicating high computational efficiency.
• The use of factorized 3D convolutions significantly reduced the number of training parameters while maintaining high registration accuracy.