[Paper Review] Deep Learning in Medical Ultrasound Image Segmentation: a Review

Ziyang Wang|arXiv (Cornell University)|Feb 18, 2020

Radiomics and Machine Learning in Medical Imaging36 references18 citations

TL;DR

This review systematically categorizes deep learning methods for medical ultrasound image segmentation into six groups—FCN, Encoder-Decoder, RNN, GAN, Weakly Supervised Learning, and Deep Reinforcement Learning—providing detailed analysis of representative models, evaluation protocols, datasets, and performance. It highlights the dominance of 2D FCN and U-Net architectures, identifies challenges in data annotation and model efficiency, and outlines future research directions including 3D segmentation, lightweight models, and multi-modal learning.

ABSTRACT

Applying machine learning technologies, especially deep learning, into medical image segmentation is being widely studied because of its state-of-the-art performance and results. It can be a key step to provide a reliable basis for clinical diagnosis, such as 3D reconstruction of human tissues, image-guided interventions, image analyzing and visualization. In this review article, deep-learning-based methods for ultrasound image segmentation are categorized into six main groups according to their architectures and training at first. Secondly, for each group, several current representative algorithms are selected, introduced, analyzed and summarized in detail. In addition, common evaluation methods for image segmentation and ultrasound image segmentation datasets are summarized. Further, the performance of the current methods and their evaluations are reviewed. In the end, the challenges and potential research directions for medical ultrasound image segmentation are discussed.

Motivation & Objective

• To provide a comprehensive, up-to-date review of deep learning techniques specifically for medical ultrasound image segmentation.
• To categorize existing deep learning methods into six distinct groups based on architecture and training strategy.
• To summarize common evaluation metrics, public datasets, and reported performance across different methods.
• To identify key challenges in data annotation, model efficiency, and image heterogeneity in ultrasound imaging.
• To outline future research directions, including 3D segmentation, lightweight models, and multi-modal transfer learning.

Proposed method

• Categorizes deep learning-based ultrasound segmentation methods into six groups: Fully Convolutional Networks (FCN), Encoder-Decoder networks, Recurrent Neural Networks (RNN), Generative Adversarial Networks (GAN), Weakly Supervised Learning (WSL), and Deep Reinforcement Learning (DRL).
• Reviews representative models in each category, such as CFS-FCN, DF-FCN, and DSN-OB for FCN; U-Net and its variants for Encoder-Decoder; and specialized models like FickleNet and DRL-based approaches for specific tasks.
• Analyzes architectural innovations such as skip connections, attention mechanisms, multi-scale feature fusion, and 3D context integration via direction-fused feature maps.
• Evaluates models using standard metrics including Dice Similarity Coefficient (DSC), Intersection over Union (IOU), Jaccard Index, and accuracy, with results reported on public datasets like PASCAL VOC 2012, Shenzhen, and Automated Cardiac Diagnosis Challenge.
• Discusses training strategies such as multi-stage incremental learning, unsupervised domain adaptation, and self-play in DRL for improved generalization.
• Reviews the use of 2D pre-trained models to enhance 3D ultrasound segmentation through cross-sectional reorganization and 3D feature fusion.

Experimental results

Research questions

• RQ1 How do different deep learning architectures—such as FCN, U-Net, GANs, and RNNs—perform in segmenting medical ultrasound images across various anatomical structures?
• RQ2 What are the most effective evaluation metrics and benchmark datasets used to validate ultrasound image segmentation models?
• RQ3 What are the key limitations of current deep learning models in ultrasound segmentation, particularly regarding data annotation cost and model efficiency?
• RQ4 How can weakly supervised and self-supervised learning approaches reduce reliance on expensive pixel-level annotations in ultrasound imaging?
• RQ5 What future research directions, such as 3D segmentation, lightweight models, and multi-modal learning, are most promising for advancing clinical deployment?

Key findings

• Nearly 80% of deep learning models for ultrasound segmentation are based on 2D fully convolutional networks or Encoder-Decoder architectures, indicating a strong reliance on 2D approaches despite the availability of 3D data.
• The DSN-OB model achieved promising results in blood vessel and lesion segmentation by combining coarse and fine-resolution supervision with a fusion layer to reduce boundary ambiguity.
• The Direction-Fused FCN (DF-FCN) model improved 3D context modeling by fusing feature maps from three orthogonal directions, enhancing segmentation accuracy in catheter detection tasks.
• Weakly supervised learning methods such as those using bounding box or line annotations achieved moderate performance, with IOU scores of 21.61–24.61 on the MC dataset and DSC of 84.72 on the Spinal Cord Gray Matter Challenge.
• Deep Reinforcement Learning methods demonstrated strong performance in prostate segmentation, achieving an area overlap of 0.9096 on a small dataset of 60 images.
• Despite high performance on benchmark datasets, models often fail to generalize due to the heterogeneous appearance of ultrasound images caused by organ depth, tissue properties, and operator variability.