[Paper Review] Performance Deterioration of Deep Learning Models after Clinical Deployment: A Case Study with Auto-segmentation for Definitive Prostate Cancer Radiotherapy

Biling Wang, Michael Dohopolski+11 more|arXiv (Cornell University)|Oct 11, 2022

Prostate Cancer Diagnosis and Treatment4 citations

TL;DR

This study investigates performance degradation in a deep learning-based auto-segmentation model for prostate cancer radiotherapy after clinical deployment. Using a UNet model trained on 2006–2011 data and tested on 2012–2022 data from 1,328 patients, it reveals significant DSC decline for prostate and rectum contours post-2015, driven by evolving clinical practices such as hydrogel spacers, slice thickness, and IV contrast use, highlighting the need for ongoing model monitoring and adaptation.

ABSTRACT

We evaluated the temporal performance of a deep learning (DL) based artificial intelligence (AI) model for auto segmentation in prostate radiotherapy, seeking to correlate its efficacy with changes in clinical landscapes. Our study involved 1328 prostate cancer patients who underwent definitive radiotherapy from January 2006 to August 2022 at the University of Texas Southwestern Medical Center. We trained a UNet based segmentation model on data from 2006 to 2011 and tested it on data from 2012 to 2022 to simulate real world clinical deployment. We measured the model performance using the Dice similarity coefficient (DSC), visualized the trends in contour quality using exponentially weighted moving average (EMA) curves. Additionally, we performed Wilcoxon Rank Sum Test to analyze the differences in DSC distributions across distinct periods, and multiple linear regression to investigate the impact of various clinical factors. The model exhibited peak performance in the initial phase (from 2012 to 2014) for segmenting the prostate, rectum, and bladder. However, we observed a notable decline in performance for the prostate and rectum after 2015, while bladder contour quality remained stable. Key factors that impacted the prostate contour quality included physician contouring styles, the use of various hydrogel spacer, CT scan slice thickness, MRI-guided contouring, and using intravenous (IV) contrast. Rectum contour quality was influenced by factors such as slice thickness, physician contouring styles, and the use of various hydrogel spacers. The bladder contour quality was primarily affected by using IV contrast. This study highlights the challenges in maintaining AI model performance consistency in a dynamic clinical setting. It underscores the need for continuous monitoring and updating of AI models to ensure their ongoing effectiveness and relevance in patient care.

Motivation & Objective

• To evaluate the long-term performance stability of a deep learning auto-segmentation model after clinical deployment in prostate cancer radiotherapy.
• To identify clinical and imaging factors contributing to performance deterioration in AI-based contouring over time.
• To assess temporal changes in Dice similarity coefficient (DSC) for prostate, rectum, and bladder structures across a 16-year period.
• To investigate the impact of evolving clinical practices—such as hydrogel spacers, slice thickness, and IV contrast—on model performance.
• To advocate for continuous model monitoring and retraining protocols to maintain AI efficacy in dynamic clinical environments.

Proposed method

• A UNet-based deep learning model was trained on 1,328 prostate cancer patient scans from 2006 to 2011 for auto-segmentation of prostate, rectum, and bladder.
• Model performance was evaluated on test data from 2012 to 2022 using the Dice similarity coefficient (DSC) as the primary metric.
• Exponentially weighted moving average (EMA) curves were used to visualize temporal trends in contour quality across time periods.
• Wilcoxon Rank Sum Test was applied to detect statistically significant differences in DSC distributions across distinct time intervals.
• Multiple linear regression was used to quantify the impact of clinical factors—such as slice thickness, hydrogel use, IV contrast, and physician contouring styles—on DSC values.
• The study simulated real-world deployment by training on early data and testing on progressively later data, reflecting clinical evolution.

Experimental results

Research questions

• RQ1 How does the performance of a deep learning auto-segmentation model for prostate cancer radiotherapy change over time after clinical deployment?
• RQ2 Which clinical factors are most strongly associated with performance deterioration in AI-based contouring of prostate, rectum, and bladder?
• RQ3 Does the use of hydrogel spacers, varying CT slice thickness, or IV contrast significantly affect model accuracy over time?
• RQ4 To what extent do physician contouring styles influence the model’s DSC performance across different time periods?
• RQ5 Can temporal trends in DSC be reliably tracked using EMA curves and statistical tests to detect model degradation?

Key findings

• The model exhibited peak performance from 2012 to 2014, with the highest DSC values for prostate, rectum, and bladder structures.
• A significant decline in DSC was observed for the prostate and rectum after 2015, while bladder DSC remained stable over time.
• Prostate contour quality was most affected by physician contouring styles, use of hydrogel spacers, CT slice thickness, and IV contrast use.
• Rectum contour quality was significantly influenced by slice thickness, physician contouring styles, and hydrogel spacer use.
• Bladder contour quality was primarily impacted by the use of intravenous (IV) contrast, with no significant trend over time otherwise.
• The study demonstrates that evolving clinical practices—not just data distribution shifts—can drive performance degradation in deployed AI models.