[Paper Review] Large-Scale Multi-Center CT and MRI Segmentation of Pancreas with Deep Learning

Zheyuan Zhang, Elif Keleş+36 more|arXiv (Cornell University)|May 20, 2024

Hydraulic Fracturing and Reservoir Analysis4 citations

TL;DR

This paper proposes PanSegNet, a deep learning model combining nnUNet and a Transformer with linear attention for robust, multi-center, multi-modality pancreas segmentation on CT and MRI. It achieves state-of-the-art Dice scores of 88.3% (CT), 85.0% (T1W MRI), and 86.3% (T2W MRI), with high volume prediction accuracy and strong cross-center generalization.

ABSTRACT

Automated volumetric segmentation of the pancreas on cross-sectional imaging is needed for diagnosis and follow-up of pancreatic diseases. While CT-based pancreatic segmentation is more established, MRI-based segmentation methods are understudied, largely due to a lack of publicly available datasets, benchmarking research efforts, and domain-specific deep learning methods. In this retrospective study, we collected a large dataset (767 scans from 499 participants) of T1-weighted (T1W) and T2-weighted (T2W) abdominal MRI series from five centers between March 2004 and November 2022. We also collected CT scans of 1,350 patients from publicly available sources for benchmarking purposes. We developed a new pancreas segmentation method, called PanSegNet, combining the strengths of nnUNet and a Transformer network with a new linear attention module enabling volumetric computation. We tested PanSegNet's accuracy in cross-modality (a total of 2,117 scans) and cross-center settings with Dice and Hausdorff distance (HD95) evaluation metrics. We used Cohen's kappa statistics for intra and inter-rater agreement evaluation and paired t-tests for volume and Dice comparisons, respectively. For segmentation accuracy, we achieved Dice coefficients of 88.3% (std: 7.2%, at case level) with CT, 85.0% (std: 7.9%) with T1W MRI, and 86.3% (std: 6.4%) with T2W MRI. There was a high correlation for pancreas volume prediction with R^2 of 0.91, 0.84, and 0.85 for CT, T1W, and T2W, respectively. We found moderate inter-observer (0.624 and 0.638 for T1W and T2W MRI, respectively) and high intra-observer agreement scores. All MRI data is made available at https://osf.io/kysnj/. Our source code is available at https://github.com/NUBagciLab/PaNSegNet.

Motivation & Objective

• Address the critical scarcity of publicly available, high-quality pancreatic MRI datasets for deep learning.
• Develop a generalized deep learning model capable of accurate pancreas segmentation across diverse CT and MRI scans from multiple medical centers.
• Enable robust, automated volumetric segmentation of the pancreas to support diagnosis, follow-up, and surgical planning in pancreatic diseases.
• Improve cross-modality and cross-center generalization of segmentation models despite domain shifts in imaging protocols and scanner vendors.
• Foster open science by releasing a large-scale, multi-center MRI dataset and the PanSegNet source code to accelerate research in medical image analysis.

Proposed method

• Proposed PanSegNet, a hybrid deep learning architecture integrating the strengths of nnUNet for 3D U-Net-based segmentation and a Transformer with a novel linear attention mechanism for long-range context modeling.
• Designed a linear attention module to enable efficient volumetric computation, reducing memory and computational costs while maintaining performance on 3D medical images.
• Trained and validated the model on a large-scale, multi-center dataset of 767 MRI scans (385 T1W, 382 T2W) from five institutions and 1,350 CT scans from public sources.
• Applied extensive data preprocessing and harmonization to mitigate intensity and sequence variations across centers and scanner types.
• Evaluated performance using Dice coefficient, HD95, intra- and inter-rater agreement (Cohen’s kappa), and paired t-tests for volume and Dice comparisons.
• Conducted cross-modality and cross-center evaluations to test generalization, including external validation on datasets like AMOS, WORD, and BTCV.

Experimental results

Research questions

• RQ1 Can a deep learning model achieve high-accuracy pancreas segmentation across diverse CT and MRI scans from multiple medical centers with varying protocols and scanner vendors?
• RQ2 How does the proposed PanSegNet model perform in cross-modality (CT vs. MRI) and cross-center generalization settings compared to existing methods?
• RQ3 To what extent does the model’s performance degrade due to domain shifts in MRI and CT imaging data, and can it maintain high accuracy despite such shifts?
• RQ4 What is the correlation between predicted and ground-truth pancreas volumes, and how reliable is the automated volumetry for clinical applications?
• RQ5 Can the integration of a linear attention mechanism in a 3D U-Net framework improve segmentation accuracy and efficiency on large-scale volumetric medical images?

Key findings

• PanSegNet achieved a Dice coefficient of 88.3% (±7.2%) on CT scans, 85.0% (±7.9%) on T1-weighted MRI, and 86.3% (±6.4%) on T2-weighted MRI, demonstrating state-of-the-art performance.
• The model showed strong pancreas volume prediction accuracy with R² values of 0.91 (CT), 0.84 (T1W MRI), and 0.85 (T2W MRI), indicating high correlation with ground truth.
• Inter-rater agreement was moderate (kappa = 0.624 for T1W, 0.638 for T2W MRI), while intra-rater agreement was high, indicating reliable manual annotations.
• External validation revealed domain shifts, particularly in Centers #3, #4, and #5, suggesting performance degradation across unseen centers, highlighting the need for domain generalization.
• The study released the largest publicly available multi-center MRI dataset for pancreas segmentation (385 T1W, 382 T2W scans) and open-sourced PanSegNet code to support future research.
• Despite limitations in prospective clinical validation and scope (only pancreas segmentation), the model demonstrates strong potential for clinical use in diabetes, chronic pancreatitis, and cancer follow-up.