[Paper Review] Whole-body tumor segmentation of 18F -FDG PET/CT using a cascaded and ensembled convolutional neural networks

Ludovic Sibille, Xinrui Zhan+1 more|arXiv (Cornell University)|Oct 14, 2022

Medical Imaging Techniques and Applications4 citations

TL;DR

This study proposes a cascaded and ensembled 3D convolutional neural network for whole-body tumor segmentation in 18F-FDG PET/CT scans, using a coarse-to-fine approach with a 6mm-resolution initial segmentation followed by a residual-based refiner to restore original resolution. The method achieved a dice score of 0.68 on 84 test cases and showed strong correlation (R² = 0.969) between automatic and manual metabolic tumor volume measurements.

ABSTRACT

Background: A crucial initial processing step for quantitative PET/CT analysis is the segmentation of tumor lesions enabling accurate feature ex-traction, tumor characterization, oncologic staging, and image-based therapy response assessment. Manual lesion segmentation is however associated with enormous effort and cost and is thus infeasible in clinical routine. Goal: The goal of this study was to report the performance of a deep neural network designed to automatically segment regions suspected of cancer in whole-body 18F-FDG PET/CT images in the context of the AutoPET challenge. Method: A cascaded approach was developed where a stacked ensemble of 3D UNET CNN processed the PET/CT images at a fixed 6mm resolution. A refiner network composed of residual layers enhanced the 6mm segmentation mask to the original resolution. Results: 930 cases were used to train the model. 50% were histologically proven cancer patients and 50% were healthy controls. We obtained a dice=0.68 on 84 stratified test cases. Manual and automatic Metabolic Tumor Volume (MTV) were highly correlated (R2 = 0.969,Slope = 0.947). Inference time was 89.7 seconds on average. Conclusion: The proposed algorithm accurately segmented regions suspicious for cancer in whole-body 18F -FDG PET/CT images.

Motivation & Objective

• To develop an automated deep learning method for whole-body tumor segmentation in 18F-FDG PET/CT scans to overcome the limitations of manual segmentation.
• To address the high variability in FDG uptake patterns across normal tissues and tumors, including low-avidity lesions and artifacts.
• To improve segmentation accuracy and robustness through a cascaded architecture combining ensemble learning and resolution refinement.
• To achieve clinically viable inference speed while maintaining high performance on diverse lesion types and patient populations.

Proposed method

• A cascaded two-stage framework was employed: first, a stacked ensemble of four customized 3D U-Net models processed PET/CT images at a fixed 6mm isotropic resolution to generate a coarse segmentation mask.
• The ensemble model used large kernel sizes (9³) in the bottleneck to capture long-range dependencies and employed leaky ReLU activations with instance normalization.
• A residual-based refiner network was trained on the original-resolution images and the 6mm segmentation mask to refine the output to full resolution, improving spatial accuracy.
• Data augmentation included random flips, affine transformations, and PET-specific augmentations (blur, brightness, contrast, gamma), with additional random spacing for the refiner to ensure resolution independence.
• The model was trained using a combination of dice loss and cross-entropy loss, with deep supervision and gradient clipping to stabilize training.
• Post-processing included connected component analysis and feature aggregation, though sequence models (LSTM/GRU) were ultimately excluded due to minimal performance gains.

Experimental results

Research questions

• RQ1 Can a cascaded and ensembled 3D CNN architecture achieve robust whole-body tumor segmentation across diverse FDG-PET/CT appearances, including low-avidity and disseminated lesions?
• RQ2 How does combining multiple 3D U-Net models with shared architecture and ensemble averaging improve segmentation stability and performance?
• RQ3 To what extent does a resolution-refining network enhance segmentation accuracy when restoring from coarse 6mm predictions to original resolution?
• RQ4 How do different loss functions and data augmentation strategies impact model generalization and robustness to lesion variability?
• RQ5 Can the automated segmentation method achieve high concordance with manual expert delineations in metabolic tumor volume (MTV) estimation?

Key findings

• The final ensemble model achieved a dice score of 0.6823 on the 84-stratified test cases, with a dice of 0.6823 for cases with foreground lesions.
• The correlation between automatically and manually segmented metabolic tumor volumes (MTV) was very high, with R² = 0.969 and slope = 0.947.
• False positive volume was reduced to 11.59 mL (in voxels) and false negative volume to 19.59 mL in the ensemble model, indicating effective balance between sensitivity and specificity.
• The full pipeline, including the refiner, required an average of 89.7 seconds per case on a single V100 GPU, with no optimization for inference speed.
• The model demonstrated robustness to normal physiological uptake patterns (e.g., brown fat, bowel, kidneys) and pathological variations, including low-avidity and bulky lesions.
• The use of combined dice and cross-entropy loss, along with deep supervision and gradient clipping, significantly improved training stability and final performance.