[Paper Review] A Robust Ensemble Algorithm for Ischemic Stroke Lesion Segmentation: Generalizability and Clinical Utility Beyond the ISLES Challenge

Ezequiel de la Rosa, Mauricio Reyes+56 more|arXiv (Cornell University)|Mar 28, 2024

Acute Ischemic Stroke ManagementMedicine8 citations

TL;DR

The authors build a robust ensemble of top ISLES’22 submissions to detect and segment ischemic stroke lesions on diffusion-weighted MRI, achieving state-of-the-art accuracy that generalizes across centers, lesion sizes, and stroke patterns, and shows clinical relevance beyond the challenge.

ABSTRACT

Diffusion-weighted MRI (DWI) is essential for stroke diagnosis, treatment decisions, and prognosis. However, image and disease variability hinder the development of generalizable AI algorithms with clinical value. We address this gap by presenting a novel ensemble algorithm derived from the 2022 Ischemic Stroke Lesion Segmentation (ISLES) challenge. ISLES'22 provided 400 patient scans with ischemic stroke from various medical centers, facilitating the development of a wide range of cutting-edge segmentation algorithms by the research community. Through collaboration with leading teams, we combined top-performing algorithms into an ensemble model that overcomes the limitations of individual solutions. Our ensemble model achieved superior ischemic lesion detection and segmentation accuracy on our internal test set compared to individual algorithms. This accuracy generalized well across diverse image and disease variables. Furthermore, the model excelled in extracting clinical biomarkers. Notably, in a Turing-like test, neuroradiologists consistently preferred the algorithm's segmentations over manual expert efforts, highlighting increased comprehensiveness and precision. Validation using a real-world external dataset (N=1686) confirmed the model's generalizability. The algorithm's outputs also demonstrated strong correlations with clinical scores (admission NIHSS and 90-day mRS) on par with or exceeding expert-derived results, underlining its clinical relevance. This study offers two key findings. First, we present an ensemble algorithm (https://github.com/Tabrisrei/ISLES22_Ensemble) that detects and segments ischemic stroke lesions on DWI across diverse scenarios on par with expert (neuro)radiologists. Second, we show the potential for biomedical challenge outputs to extend beyond the challenge's initial objectives, demonstrating their real-world clinical applicability.

Motivation & Objective

• Motivate the need for generalizable AI for ischemic stroke lesion segmentation across diverse datasets.
• Leverage ISLES’22 challenge results to build a robust ensemble that overcomes individual method biases.
• Demonstrate generalization to unseen centers, varying lesion sizes, and different stroke patterns and vascular territories.
• Assess clinical utility through correlations with NIHSS and 90-day mRS and a Turing-like reader preference test.

Proposed method

• Construct an ensemble from top ISLES’22 teams (SEALS, NVAUTO, SWAN).
• Train and validate using ISLES’22 data with held-out test set and external real-world dataset (N=1686).
• Preprocess images (resample to 1x1x1 mm3, z-score normalization) and use DWI/ADC/FLAIR inputs; apply cross-validation and model ensembling.
• Evaluate with Dice, lesion-wise F1, absolute volume difference (AVD), and absolute lesion count difference (ALD).
• Assess stroke subgroups (pattern and vascular territory) and perform subjective Turing-like neuroradiologist evaluation.
• Correlate segmentation outputs with clinical scores (admission NIHSS, 90-day mRS).

Figure 1: Overview of the ISLES’22 challenge and post-challenge experimental design, including the developed algorithmic solutions. A) Challenge and post-challenge phases and datasets. B) Summary of algorithmic solutions stratified by network architecture, loss function, and input modalities. C) Cha

Experimental results

Research questions

• RQ1 Can a challenge-derived ensemble generalize to unseen imaging centers and real-world data?
• RQ2 How does the ensemble perform across lesion size, stroke phase, and stroke pattern subgroups?
• RQ3 Can the ensemble identify affected vascular territories and stroke sub-types with high accuracy?
• RQ4 Do clinicians prefer the ensemble’s segmentations over manual expert delineations in a Turing-like test?
• RQ5 Do segmentation outputs correlate with key clinical outcomes (NIHSS, 90-day mRS) on external data?

Key findings

• The ensemble achieved a top performance on unseen ISLES’22 test data (median Dice 0.82, median lesion-wise F1 0.86).
• External real-world data (N=1686) showed similar generalizability (median Dice 0.82, median lesion-wise F1 0.86).
• Lesion-size analyses showed high volumetric agreement (Pearson r = 0.98 overall; r = 0.87 for <5 ml; r = 0.90 for 5–20 ml; r = 0.96 for ≥20 ml).
• Performance was robust to center (unseen center generalization) and across stroke phases, with some Dice score differences between acute and sub-acute scans due to phase-related factors.
• The ensemble outperformed individual challenge solutions in stroke pattern classification (balanced accuracy 86.9% vs. best individual 78.9%) and vascular territory identification (balanced accuracy 97.6%).
• In a Turing-like assessment, neuroradiologists preferred the ensemble’s segmentations over manual expert delineations.

Figure 2: Performance for the participating teams in the unseen test phase of the challenge. Teams are displayed in red and in decreasing order based on their final rank. DSC: Dice Similarity Coefficient; F1 score: lesion-wise F1 score; AVD: absolute volume difference; ALD: absolute lesion count dif