[Paper Review] Multimodal Interactive Lung Lesion Segmentation: A Framework for Annotating PET/CT Images based on Physiological and Anatomical Cues

Verena Jasmin Hallitschke, Tobias Schlumberger+7 more|arXiv (Cornell University)|Jan 24, 2023

Radiomics and Machine Learning in Medical ImagingMedicine21 references3 citations

TL;DR

This paper proposes a multimodal interactive segmentation framework for PET/CT lung lesion annotation that leverages anatomical (CT) and physiological (PET) cues to reduce manual annotation effort. By using a novel ellipsoid-based user simulation and RoI-optimized geodesic distance transform, the method achieves faster, more accurate segmentation with expert users saving ~12.5% time per volume compared to baseline methods.

ABSTRACT

Recently, deep learning enabled the accurate segmentation of various diseases in medical imaging. These performances, however, typically demand large amounts of manual voxel annotations. This tedious process for volumetric data becomes more complex when not all required information is available in a single imaging domain as is the case for PET/CT data. We propose a multimodal interactive segmentation framework that mitigates these issues by combining anatomical and physiological cues from PET/CT data. Our framework utilizes the geodesic distance transform to represent the user annotations and we implement a novel ellipsoid-based user simulation scheme during training. We further propose two annotation interfaces and conduct a user study to estimate their usability. We evaluated our model on the in-domain validation dataset and an unseen PET/CT dataset. We make our code publicly available: https://github.com/verena-hallitschke/pet-ct-annotate.

Motivation & Objective

• Address the high annotation burden in 3D medical image segmentation, especially for multimodal PET/CT data where lesions are not clearly visible in a single modality.
• Reduce reliance on large-scale voxel-level annotations by enabling interactive segmentation with minimal user input (e.g., scribbles).
• Improve segmentation accuracy and efficiency by integrating both PET (metabolic activity) and CT (anatomical detail) modalities during annotation.
• Evaluate the impact of different user interface designs—simultaneous vs. sequential modality viewing—on annotation speed and quality.
• Demonstrate generalization of the model to unseen datasets, showing its robustness beyond the training distribution.

Proposed method

• Adapt the DeepIGeoS model to handle dual-modal inputs (PET and CT) by sharing the same CNN architecture for both proposal (P-Net) and refinement (R-Net) networks.
• Introduce a novel ellipsoid-based user simulation scheme to generate realistic foreground and background scribbles during training, improving model generalization.
• Encode user interactions using a region-of-interest (RoI)-based geodesic distance transform to reduce computational cost while preserving spatial context.
• Implement two user interface designs: (1) side-by-side display of PET and CT for simultaneous viewing, and (2) toggleable modality view for focused attention.
• Use 3D Slicer with MONAI Label to build an interactive annotation plugin that enables real-time model inference and iterative refinement.
• Train the model end-to-end on the AutoPET dataset and evaluate on both in-domain and out-of-domain (Lung-PET-CT-Dx) datasets to assess generalization.

Fig. 1 : We introduce an interactive segmentation framework containing paired multimodal views (left). We propose a novel ellipsoid-based user simulation scheme during training (middle) and we use only the RoI when computing the geodesic transform (right) to save computational time.

Experimental results

Research questions

• RQ1 Does displaying both PET and CT modalities simultaneously in the user interface lead to faster and more accurate lung lesion segmentation compared to sequential viewing?
• RQ2 Can a multimodal interactive segmentation framework significantly reduce the time and effort required for annotating lung lesions in PET/CT volumes?
• RQ3 How effective is the proposed ellipsoid-based user simulation scheme in improving model robustness during training without real user interactions?
• RQ4 To what extent can the model improve initial deep learning predictions on unseen PET/CT datasets through interactive refinement?
• RQ5 How does the integration of physiological (PET) and anatomical (CT) cues enhance segmentation performance compared to single-modality approaches?

Key findings

• The two-view interface (simultaneous PET and CT display) reduced annotation time by approximately 12.5% compared to GraphCut and single-view setups, confirming faster annotation with multimodal visibility.
• All users achieved acceptable Dice scores (>0.7) when using the proposed method with the two-view interface, while GraphCut failed to produce usable segmentations, resulting in near-zero Dice scores.
• The single-view interface yielded similar Dice scores to the two-view interface for most users, but annotation was significantly slower, indicating that simultaneous viewing improves efficiency.
• The model successfully improved initial P-Net predictions on an unseen dataset (Lung-PET-CT-Dx), demonstrating strong generalization capability across different data distributions.
• In post-study feedback, all experts found the multimodal interface helpful and reported high satisfaction with the speed and quality of the model’s initial proposals.
• The framework achieved state-of-the-art performance in interactive segmentation for PET/CT by combining physiological and anatomical cues through optimized user interaction modeling.

Fig. 2 : The workflow of our proposed method. Blue boxes indicate the segmentation models which are adapted from Wang et al. [ 6 ] to multimodal inputs.