[Paper Review] VoxResNet: Deep Voxelwise Residual Networks for Volumetric Brain Segmentation

Hao Chen, Qi Dou+2 more|arXiv (Cornell University)|Aug 21, 2016

Advanced Neural Network Applications34 references115 citations

TL;DR

VoxResNet extends residual learning to 3D for volumetric brain segmentation and boosts performance by fusing multi-modality data and auto-context information.

ABSTRACT

Recently deep residual learning with residual units for training very deep neural networks advanced the state-of-the-art performance on 2D image recognition tasks, e.g., object detection and segmentation. However, how to fully leverage contextual representations for recognition tasks from volumetric data has not been well studied, especially in the field of medical image computing, where a majority of image modalities are in volumetric format. In this paper we explore the deep residual learning on the task of volumetric brain segmentation. There are at least two main contributions in our work. First, we propose a deep voxelwise residual network, referred as VoxResNet, which borrows the spirit of deep residual learning in 2D image recognition tasks, and is extended into a 3D variant for handling volumetric data. Second, an auto-context version of VoxResNet is proposed by seamlessly integrating the low-level image appearance features, implicit shape information and high-level context together for further improving the volumetric segmentation performance. Extensive experiments on the challenging benchmark of brain segmentation from magnetic resonance (MR) images corroborated the efficacy of our proposed method in dealing with volumetric data. We believe this work unravels the potential of 3D deep learning to advance the recognition performance on volumetric image segmentation.

Motivation & Objective

• Motivate robust volumetric brain segmentation using deep residual learning in 3D.
• Propose VoxResNet, a deep voxelwise residual network for volumetric segmentation.
• Introduce auto-context VoxResNet to integrate low-, mid-, and high-level contextual information.
• Demonstrate benefits of multi-modality data fusion in MR brain segmentation.

Proposed method

• Extend 2D deep residual networks to a 3D VoxResNet with 25 volumetric conv layers and 4 deconvolution layers.
• Use small 3x3x3 kernels with stride-2 convs to achieve a large receptive field.
• Incorporate post-activation residual units with skip connections for stable deep training.
• Integrate multi-modality inputs (e.g., T1, T1-IR, T2-FLAIR) by concatenation and joint learning.
• Apply deep supervision with four auxiliary classifiers (C1–C4) to fuse multi-scale context.
• Adopt an auto-context scheme where initial VoxResNet outputs guide a second-stage Auto-context VoxResNet for refinement.
• Train with a loss combining regularization and voxel-wise cross-entropy, including auxiliary losses with weighted emphasis.

Experimental results

Research questions

• RQ1 Can a 3D deep residual network (VoxResNet) effectively learn volumetric brain segmentation from MR images?
• RQ2 Does multi-modality data fusion improve segmentation accuracy in a 3D context?
• RQ3 Does integrating auto-context information further enhance segmentation performance?
• RQ4 What is the empirical impact of deep supervision on 3D volumetric segmentation accuracy?

Key findings

• VoxResNet with multi-modality inputs outperforms single-modality baselines on brain tissue segmentation.
• Auto-context integration further improves Dice coefficients (DC) and reduces segmentation errors.
• The All modality setup achieves high DC values across tissues (GM, WM, CSF) and competitive Hausdorff distances and absolute volume differences.
• Auto-context VoxResNet yields additional gains over VoxResNet alone in validation tests.
• On the MRBrainS benchmark, the proposed method ranked among the top entries, illustrating strong performance in volumetric brain segmentation.