[Paper Review] Medical Image Segmentation Review: The success of U-Net

Reza Azad, Ehsan Khodapanah Aghdam+8 more|arXiv (Cornell University)|Nov 27, 2022

Brain Tumor Detection and Classification96 citations

TL;DR

This paper surveys U-Net and its variants for medical image segmentation, proposes a six-category taxonomy, and offers practical resources including code, pre-trained models, and an online paper list.

ABSTRACT

Automatic medical image segmentation is a crucial topic in the medical domain and successively a critical counterpart in the computer-aided diagnosis paradigm. U-Net is the most widespread image segmentation architecture due to its flexibility, optimized modular design, and success in all medical image modalities. Over the years, the U-Net model achieved tremendous attention from academic and industrial researchers. Several extensions of this network have been proposed to address the scale and complexity created by medical tasks. Addressing the deficiency of the naive U-Net model is the foremost step for vendors to utilize the proper U-Net variant model for their business. Having a compendium of different variants in one place makes it easier for builders to identify the relevant research. Also, for ML researchers it will help them understand the challenges of the biological tasks that challenge the model. To address this, we discuss the practical aspects of the U-Net model and suggest a taxonomy to categorize each network variant. Moreover, to measure the performance of these strategies in a clinical application, we propose fair evaluations of some unique and famous designs on well-known datasets. We provide a comprehensive implementation library with trained models for future research. In addition, for ease of future studies, we created an online list of U-Net papers with their possible official implementation. All information is gathered in https://github.com/NITR098/Awesome-U-Net repository.

Motivation & Objective

• Provide a comprehensive overview of U-Net-based medical image segmentation methods and their modalities.
• Propose a taxonomy to categorize U-Net variants based on architectural design changes.
• Offer practical guidance with datasets, loss functions, evaluation metrics, and comparative evaluations.
• Present an implementation library with trained models and an online list of U-Net papers and implementations.

Proposed method

• Introduce a six-category taxonomy for U-Net variants: Skip Connection Enhancements, Backbone Design Enhancements, Bottleneck Enhancements, Transformers, Rich Representation Enhancements, and Probabilistic Design.
• Detail 2D and 3D U-Net architectures and their applicability to medical imaging modalities.
• Survey and discuss numerous extensions focusing on skip connections, feature map processing, and attention mechanisms.
• Provide comparative experiments on popular datasets to evaluate design choices.
• Offer practical resources including a GitHub repository with implementations and pre-trained weights, and an online list of U-Net papers.
• Organize the discussion to aid both vendors and researchers in selecting appropriate U-Net variants.

Figure 1 : The number of research works published in the past decade using the U-Net model as their baseline to address various medical image analysis challenges. The visualization shows sumptuous attention from the research/industry community for this architecture, particularly the segmentation tas

Experimental results

Research questions

• RQ1 What design modifications to U-Net most effectively improve medical image segmentation across modalities?
• RQ2 How do skip connections, backbones, bottlenecks, transformers, and probabilistic designs impact performance and practicality?
• RQ3 What datasets and evaluation practices best reflect real-world clinical utility for U-Net variants?
• RQ4 What resources (code, weights, papers) best support researchers deploying U-Net models in practice.

Key findings

• The paper catalogs U-Net variants into six practical categories, clarifying how each modification targets segmentation challenges.
• It reviews literature on more than 100 U-Net-based segmentation methods up to September 2022.
• The study includes comparative experiments on well-known datasets to illustrate performance trends across designs.
• It provides a comprehensive implementation library with trained models and an online list of U-Net papers with available implementations.
• The work emphasizes clinical relevance, data modalities, loss functions, and evaluation metrics in assessing U-Net extensions.

Figure 2 : The proposed U-Net taxonomy categorizes different extensions of the U-Net model based on their underlying design idea. More specifically, our taxonomy takes into account the modular design of the U-Net model and shows where the improvement happens (e.g., skip connection). Due to the clari