[Paper Review] Combining Fully Convolutional and Recurrent Neural Networks for 3D Biomedical Image Segmentation

Jianxu Chen, Lin Yang+3 more|arXiv (Cornell University)|Sep 5, 2016

Cell Image Analysis Techniques19 references220 citations

TL;DR

Introduces a two-component framework that combines a k-U-Net FCN for intra-slice feature extraction with a Bi-Directional Convolutional LSTM (BDC-LSTM) to capture inter-slice 3D context, improving 3D biomedical image segmentation especially with anisotropic data.

ABSTRACT

Segmentation of 3D images is a fundamental problem in biomedical image analysis. Deep learning (DL) approaches have achieved state-of-the-art segmentation perfor- mance. To exploit the 3D contexts using neural networks, known DL segmentation methods, including 3D convolution, 2D convolution on planes orthogonal to 2D image slices, and LSTM in multiple directions, all suffer incompatibility with the highly anisotropic dimensions in common 3D biomedical images. In this paper, we propose a new DL framework for 3D image segmentation, based on a com- bination of a fully convolutional network (FCN) and a recurrent neural network (RNN), which are responsible for exploiting the intra-slice and inter-slice contexts, respectively. To our best knowledge, this is the first DL framework for 3D image segmentation that explicitly leverages 3D image anisotropism. Evaluating using a dataset from the ISBI Neuronal Structure Segmentation Challenge and in-house image stacks for 3D fungus segmentation, our approach achieves promising results comparing to the known DL-based 3D segmentation approaches.

Motivation & Objective

• Address the challenge of exploiting 3D context in highly anisotropic biomedical images.
• Propose a framework that separates intra-slice feature extraction from inter-slice context modeling.
• Improve 3D segmentation by leveraging both 2D multi-scale FCN and stacked BDC-LSTM to capture 3D context.

Proposed method

• Propose a two-component architecture: k U-Net (multi-scale 2D FCN) for intra-slice feature extraction and BDC-LSTM (Bi-Directional Convolutional LSTM) for inter-slice 3D context integration.
• Introduce k U-Net: a sequence of submodule U-Nets operating at progressively coarser image scales with information propagation from coarser to finer scales.
• Introduce BDC-LSTM: two CLSTM layers operating in opposite z-directions, concatenating contexts to form 3D features, with possible deep stacking and integration of max-pooling and deconvolution to build a hierarchy of contextual features.
• Combine k U-Net and BDC-LSTM by first extracting 2D feature maps per slice, then feeding a sequence of these maps into BDC-LSTM to produce 3D segmentation probability maps.
• Training strategies include end-to-end or decoupled training, with data augmentation, dropout, and weighted cross-entropy loss to emphasize boundaries or regions of interest.

Experimental results

Research questions

• RQ1 Can a combination of multi-scale 2D FCN (k U-Net) and inter-slice RNN (BDC-LSTM) better exploit anisotropic 3D contexts than existing 3D CNN or RNN approaches?
• RQ2 Does separating intra-slice feature extraction from inter-slice context modeling improve segmentation accuracy on anisotropic 3D biomedical datasets?
• RQ3 How does the proposed architecture compare to Pyramid-LSTM and other 3D segmentation methods across different datasets with varying z-resolution?

Key findings

• The proposed FCN+RNN framework achieves competitive or superior segmentation metrics compared to state-of-the-art 3D DL methods on two distinct datasets.
• k U-Net alone improves over standard U-Net by leveraging multi-scale intra-slice information.
• Deep BDC-LSTM, when combined with k U-Net, yields the best results among tested configurations.
• On the ISBI neuron dataset, the method with FCN+deep RNN achieves the highest V_rand (0.9753) and V_info (0.9870) with the lowest Pixel Error (0.0215).
• On the in-house 3D fungus dataset, FCN+deep RNN also outperforms baselines with V_rand 0.9753, V_info 0.9870, and Pixel Error 0.0215 (best among reported methods).
• The approach demonstrates efficiency in GPU memory compared to Pyramid-LSTM when re-implemented in the same framework.