[Paper Review] Joint Learning for Pulmonary Nodule Segmentation, Attributes and Malignancy Prediction

Botong Wu, Zhen Zhou+2 more|ArXiv.org|Feb 10, 2018

Lung Cancer Diagnosis and Treatment12 references11 citations

TL;DR

This paper proposes PN-SAMP, a 3D multi-task deep learning framework that jointly performs pulmonary nodule segmentation, attribute prediction, and malignancy classification using multi-window width/center CT inputs. By integrating segmentation, attributes, and malignancy prediction into a single end-to-end model, it achieves state-of-the-art performance—97.58% accuracy in malignancy prediction, 89.33% in attribute prediction, and 73.89% Dice score in segmentation—while enhancing interpretability for radiologists through explainable outputs.

ABSTRACT

Refer to the literature of lung nodule classification, many studies adopt Convolutional Neural Networks (CNN) to directly predict the malignancy of lung nodules with original thoracic Computed Tomography (CT) and nodule location. However, these studies cannot tell how the CNN works in terms of predicting the malignancy of the given nodule, e.g., it's hard to conclude that whether the region within the nodule or the contextual information matters according to the output of the CNN. In this paper, we propose an interpretable and multi-task learning CNN -- Joint learning for extbf{P}ulmonary extbf{N}odule extbf{S}egmentation extbf{A}ttributes and extbf{M}alignancy extbf{P}rediction (PN-SAMP). It is able to not only accurately predict the malignancy of lung nodules, but also provide semantic high-level attributes as well as the areas of detected nodules. Moreover, the combination of nodule segmentation, attributes and malignancy prediction is helpful to improve the performance of each single task. In addition, inspired by the fact that radiologists often change window widths and window centers to help to make decision on uncertain nodules, PN-SAMP mixes multiple WW/WC together to gain information for the raw CT input images. To verify the effectiveness of the proposed method, the evaluation is implemented on the public LIDC-IDRI dataset, which is one of the largest dataset for lung nodule malignancy prediction. Experiments indicate that the proposed PN-SAMP achieves significant improvement with respect to lung nodule classification, and promising performance on lung nodule segmentation and attribute learning, compared with the-state-of-the-art methods.

Motivation & Objective

• To address the lack of interpretability in existing deep learning models for lung nodule malignancy prediction, which often produce black-box decisions without clinical context.
• To improve the performance of nodule classification, segmentation, and attribute prediction by leveraging multi-task learning with shared representations.
• To enhance model interpretability by providing radiologists with semantic attributes (e.g., margin, calcification) and precise nodule segmentation alongside malignancy scores.
• To simulate radiologist practices by incorporating multiple window width/window center (WW/WC) settings to enrich input image information.
• To demonstrate that joint learning improves performance across all tasks through shared supervision and feature learning.

Proposed method

• The model uses a two-channel input where nodule patches from different WW/WC settings are stacked along the channel dimension to enrich feature representation.
• A 3D convolutional neural network is designed with shared and task-specific branches for joint learning of segmentation, attribute classification, and malignancy prediction.
• The segmentation branch is connected to the classification branch via a skip connection, enabling back-propagation of classification loss to refine segmentation.
• Multi-task learning is implemented by jointly optimizing segmentation, attribute, and malignancy prediction losses using a weighted sum of cross-entropy losses.
• The model is trained end-to-end on the LIDC-IDRI dataset using 64×64×64×2 input volumes normalized to 0.6 mm isotropic resolution.
• The framework includes a multi-scale WW/WC strategy to simulate radiologist windowing techniques, improving sensitivity to subtle nodule features.

Experimental results

Research questions

• RQ1 Can joint learning of segmentation, attributes, and malignancy prediction improve performance across all tasks compared to single-task models?
• RQ2 Does incorporating multiple window width/window center settings enhance the model’s ability to detect subtle nodule features and improve classification accuracy?
• RQ3 To what extent does multi-task training improve interpretability by providing radiologists with semantic attributes and segmentation masks?
• RQ4 How does the proposed multi-task architecture compare to state-of-the-art methods in terms of malignancy prediction, segmentation, and attribute learning on the LIDC-IDRI dataset?
• RQ5 Can the integration of nodule segmentation as a supervisory signal improve the performance of malignancy classification?

Key findings

• The proposed PN-SAMP achieved 97.58% accuracy in lung nodule malignancy prediction, outperforming all compared state-of-the-art methods.
• The model achieved 89.33% accuracy in predicting the nine high-level attributes, demonstrating strong performance in attribute learning.
• The segmentation task achieved a Dice coefficient of 73.89%, significantly outperforming the 3D-UNet baseline by 1.99% with only 1.5M parameters versus 16M.
• Multi-task learning improved performance by 3.69% in segmentation and 4.69% in classification compared to single-task baselines.
• The combination of multiple WW/WC inputs led to the best overall performance, confirming its value in enriching nodule appearance features.
• Qualitative results showed that the model's predictions for malignancy and attributes were consistent with ground truth, with most predictions within ±1 of the true rating.