[Paper Review] 3D Convolutional Neural Networks Image Registration Based on Efficient Supervised Learning from Artificial Deformations

Hessam Sokooti, Bob D. de Vos+6 more|arXiv (Cornell University)|Aug 27, 2019

Medical Image Segmentation Techniques26 references40 citations

TL;DR

This paper presents RegNet, a supervised 3D CNN-based nonrigid image registration method trained with artificial displacement vector fields (DVFs), evaluated on chest CT datasets and delivering TREs around 2–3 mm with rapid inference in a multi-stage setup.

ABSTRACT

We propose a supervised nonrigid image registration method, trained using artificial displacement vector fields (DVF), for which we propose and compare three network architectures. The artificial DVFs allow training in a fully supervised and voxel-wise dense manner, but without the cost usually associated with the creation of densely labeled data. We propose a scheme to artificially generate DVFs, and for chest CT registration augment these with simulated respiratory motion. The proposed architectures are embedded in a multi-stage approach, to increase the capture range of the proposed networks in order to more accurately predict larger displacements. The proposed method, RegNet, is evaluated on multiple databases of chest CT scans and achieved a target registration error of 2.32 $\pm$ 5.33 mm and 1.86 $\pm$ 2.12 mm on SPREAD and DIR-Lab-4DCT studies, respectively. The average inference time of RegNet with two stages is about 2.2 s.

Motivation & Objective

• Motivate and enable robust nonrigid image registration without manual voxel-wise ground truth by using artificially generated DVFs for supervision.
• Develop a multi-stage registration framework to extend capture range and better align large displacements.
• Explore and compare three network architectures (U-Net based, Multi-View, and U-Net Advanced) for efficient 3D DVF prediction.
• Assess RegNet on multiple chest CT databases (SPREAD, DIR-Lab-4DCT, DIR-Lab-COPDgene) and compare with conventional elastix-based registration.
• Demonstrate feasible inference times suitable for practical use.

Proposed method

• Generate artificial DVFs of three categories (single frequency, mixed frequency, respiratory motion) plus identity to create dense, supervised training targets.
• Augment artificial DVFs with simulated respiratory motion and intensity models to produce realistic training pairs.
• Evaluate three RegNet architectures (U-Net, Multi-View, U-Net Advanced) in a multi-stage pipeline (down to 1/4, 1/2, and full resolution) to capture large displacements.
• Train with Huber loss between predicted and ground-truth DVFs plus a bending-energy regularizer to promote smoothness.
• Measure performance using TRE against landmarks and Jacobian statistics to monitor local volume changes.

Experimental results

Research questions

• RQ1 Can densely supervised training on artificial DVFs enable accurate nonrigid registration without manual voxelwise ground truth?
• RQ2 Does a multi-stage registration pipeline improve capture range and accuracy for large displacements in chest CT registration?
• RQ3 Which RegNet architecture (U-Net, Multi-View, U-Net Advanced) offers best tradeoffs in accuracy, memory, and inference time for 3D DVF prediction?
• RQ4 How does RegNet compare to conventional elastix-based registration and other CNN-based methods on SPREAD, DIR-Lab-4DCT, and DIR-Lab-COPDgene datasets?
• RQ5 What is the impact of including respiratory motion augmentation on registration accuracy for inhale-exhale datasets?

Key findings

• RegNet achieves target registration errors of 2.32 ± 5.33 mm on SPREAD and 1.86 ± 2.12 mm on DIR-Lab-4DCT using a two-stage configuration (and related variants).
• A three-architecture comparison shows that multi-stage, patch-based designs and a U-Net style network can be effectively combined to improve capture range and accuracy.
• Multi-stage networks consistently outperform single-stage counterparts on the evaluated datasets.
• Inference times are favorable, with two-stage RegNet delivering around 2.2 seconds per subject per the reported settings.
• Training with artificial DVFs (S, M, and R categories) plus identity yields competitive TREs compared to traditional B-spline elastix registrations in several cases.