[Paper Review] Improving Deep Pancreas Segmentation in CT and MRI Images via Recurrent Neural Contextual Learning and Direct Loss Function

Jinzheng Cai, Le Lü+3 more|arXiv (Cornell University)|Jul 16, 2017

Advanced Neural Network Applications15 references130 citations

TL;DR

The paper introduces a convolutional LSTM-based contextual regularization and a Jaccard loss to directly optimize segmentation quality for pancreas in CT and MRI, achieving state-of-the-art results.

ABSTRACT

Deep neural networks have demonstrated very promising performance on accurate segmentation of challenging organs (e.g., pancreas) in abdominal CT and MRI scans. The current deep learning approaches conduct pancreas segmentation by processing sequences of 2D image slices independently through deep, dense per-pixel masking for each image, without explicitly enforcing spatial consistency constraint on segmentation of successive slices. We propose a new convolutional/recurrent neural network architecture to address the contextual learning and segmentation consistency problem. A deep convolutional sub-network is first designed and pre-trained from scratch. The output layer of this network module is then connected to recurrent layers and can be fine-tuned for contextual learning, in an end-to-end manner. Our recurrent sub-network is a type of Long short-term memory (LSTM) network that performs segmentation on an image by integrating its neighboring slice segmentation predictions, in the form of a dependent sequence processing. Additionally, a novel segmentation-direct loss function (named Jaccard Loss) is proposed and deep networks are trained to optimize Jaccard Index (JI) directly. Extensive experiments are conducted to validate our proposed deep models, on quantitative pancreas segmentation using both CT and MRI scans. Our method outperforms the state-of-the-art work on CT [11] and MRI pancreas segmentation [1], respectively.

Motivation & Objective

• Motivate accurate pancreas segmentation in CT and MRI due to small organ size and variable boundaries.
• Develop a compact CNN trained from scratch to avoid overfitting and improve efficiency.
• Incorporate contextual regularization across neighboring slices via convolutional LSTM to enforce slice-to-slice consistency.
• Introduce a segmentation-direct loss (Jaccard Loss) to optimize Jaccard Index directly.
• Evaluate on 82 CT and 79 MRI datasets with 4-fold cross-validation to demonstrate performance gains.

Proposed method

• Design a compact CNN with Scale blocks built from CBR (Convolution-BatchNorm-ReLU) units and auxiliary losses.
• Attach a CLSTM module to the CNN outputs to perform contextual learning across adjacent slices.
• Train the network end-to-end with SGD to fine-tune CNN and CLSTM jointly.
• Propose Jaccard Loss to optimize the Jaccard Index directly during training, enabling threshold-free segmentation.
• Compare JACLoss and contextual models across CT and MRI datasets to assess gains over baselines.

Experimental results

Research questions

• RQ1 Can a CNN with recurrent contextual learning across slices improve pancreas segmentation consistency compared to slice-wise 2D models?
• RQ2 Does optimizing Jaccard Loss directly lead to threshold-free, more robust segmentation than cross-entropy losses?
• RQ3 What is the impact of different CLSM configurations (64 vs 128 channels) on CT and MRI pancreas segmentation?
• RQ4 How do the proposed methods perform relative to state-of-the-art methods on NIH-CT-82 and UFL-MRI-79 datasets?
• RQ5 Is the approach generalizable to other 3D organ segmentation tasks?

Key findings

• Contextual regularization with CLSTM improves mean DSC by about 2.0 percentage points (CT-82) and 0.9 points (CT-128 variant) over non-contextual baselines.
• On MRI-79, RNN-64 improves DSC by 1.8 percentage points over JAC-64; RNN-128 achieves best results overall.
• Jaccard Loss yields the highest mean DSC across thresholds and balances foreground/background better than cross-entropy variants.
• JAC-128 achieves the best results on MRI-79, and RNN-128 achieves the best results on NIH-CT-82 among reported configurations.
• Compared to U-Net and HNN baselines, the proposed JACLoss with/contextual learning approaches offer competitive or superior Dice and Jaccard scores on both datasets.