[Paper Review] SMILE-UHURA Challenge -- Small Vessel Segmentation at Mesoscopic Scale from Ultra-High Resolution 7T Magnetic Resonance Angiograms

Soumick Chatterjee, Hendrik Mattern+46 more|arXiv (Cornell University)|Nov 14, 2024

Cerebrovascular and Carotid Artery DiseasesMedicine3 citations

TL;DR

This paper introduces the SMILE-UHURA challenge, a benchmark for segmenting mesoscopic brain vessels (100–500 µm) in ultra-high-resolution 7T MRI angiograms using deep learning. It presents a large, expert-annotated dataset and evaluates 16 methods, achieving up to 0.838 Dice score on held-out test data, demonstrating the feasibility of robust vessel segmentation at the mesoscopic scale despite high noise and low contrast.

ABSTRACT

The human brain receives nutrients and oxygen through an intricate network of blood vessels. Pathology affecting small vessels, at the mesoscopic scale, represents a critical vulnerability within the cerebral blood supply and can lead to severe conditions, such as Cerebral Small Vessel Diseases. The advent of 7 Tesla MRI systems has enabled the acquisition of higher spatial resolution images, making it possible to visualise such vessels in the brain. However, the lack of publicly available annotated datasets has impeded the development of robust, machine learning-driven segmentation algorithms. To address this, the SMILE-UHURA challenge was organised. This challenge, held in conjunction with the ISBI 2023, in Cartagena de Indias, Colombia, aimed to provide a platform for researchers working on related topics. The SMILE-UHURA challenge addresses the gap in publicly available annotated datasets by providing an annotated dataset of Time-of-Flight angiography acquired with 7T MRI. This dataset was created through a combination of automated pre-segmentation and extensive manual refinement. In this manuscript, sixteen submitted methods and two baseline methods are compared both quantitatively and qualitatively on two different datasets: held-out test MRAs from the same dataset as the training data (with labels kept secret) and a separate 7T ToF MRA dataset where both input volumes and labels are kept secret. The results demonstrate that most of the submitted deep learning methods, trained on the provided training dataset, achieved reliable segmentation performance. Dice scores reached up to 0.838 $\pm$ 0.066 and 0.716 $\pm$ 0.125 on the respective datasets, with an average performance of up to 0.804 $\pm$ 0.15.

Motivation & Objective

• To address the lack of publicly available, high-quality annotated datasets for mesoscopic vessel segmentation in 7T MRI.
• To evaluate deep learning methods on ultra-high-resolution 7T time-of-flight MRA with challenging noise and low contrast.
• To establish a benchmark for small vessel segmentation at the mesoscopic scale (100–500 µm) in human brain vasculature.
• To compare performance across diverse methods on both held-out test data and a separate secret dataset with unknown properties.
• To highlight the trade-offs between fine vessel detail capture and noise-free output quality in segmentation models.

Proposed method

• The challenge uses a 3D U-Net-based baseline and a multi-scale, multi-resolution U-Net (DS6) as reference models.
• A large, high-resolution 7T time-of-flight MRA dataset was created via automated pre-segmentation followed by extensive manual refinement by expert raters.
• Methods were trained on a public training set and evaluated on two test sets: one with known labels (held-out) and one with secret input and labels (secret dataset).
• Evaluation used Dice score, Hausdorff distance, and expert ratings on small vessel segmentation and noise-free output quality.
• The dataset includes volumes with isotropic resolutions as fine as 150 µm, enabling mesoscopic vessel visualization.
• A separate high-resolution challenge volume with double the resolution was used to test generalization and robustness.

Experimental results

Research questions

• RQ1 Can deep learning models achieve reliable segmentation of mesoscopic brain vessels in ultra-high-resolution 7T MRA despite high noise and poor contrast?
• RQ2 How do different architectures and preprocessing strategies compare in terms of Dice score and noise robustness on held-out and secret test sets?
• RQ3 What is the trade-off between capturing fine vessel details and producing clean, noise-free segmentation outputs?
• RQ4 To what extent do models generalize across datasets with varying resolution and image properties?
• RQ5 How do expert ratings correlate with quantitative metrics like Dice and Hausdorff distance in clinical relevance assessment?

Key findings

• The best-performing method achieved a Dice score of 0.838 ± 0.066 on the held-out test set, demonstrating high segmentation accuracy.
• On the secret dataset, the top method achieved a Dice score of 0.716 ± 0.125, indicating strong performance on unseen, high-noise data.
• The average Dice score across all methods was 0.804 ± 0.15, showing consistent high performance across diverse architectures.
• The Koala Manual method achieved the highest expert rating (4.0 ± 1.0) on the secret dataset for small vessel segmentation but received low noise-free scores (1.0 ± 1.5), indicating high detail at the cost of noise.
• Baseline models (UNet MSS and DS6) showed strong and consistent performance, serving as robust benchmarks across both datasets and evaluation criteria.
• Expert ratings revealed a trade-off: methods excelling in fine vessel capture (e.g., Koala Manual) often produced noisier outputs, while cleaner models sacrificed detail.