[Paper Review] A Review on Deep Learning Techniques Applied to Semantic Segmentation

Alberto García-García, Sergio Orts‐Escolano+3 more|arXiv (Cornell University)|Apr 22, 2017

Advanced Neural Network Applications96 references1,035 citations

TL;DR

This paper surveys deep learning methods for semantic segmentation, surveys datasets and challenges, and discusses performance and future directions for the field.

ABSTRACT

Image semantic segmentation is more and more being of interest for computer vision and machine learning researchers. Many applications on the rise need accurate and efficient segmentation mechanisms: autonomous driving, indoor navigation, and even virtual or augmented reality systems to name a few. This demand coincides with the rise of deep learning approaches in almost every field or application target related to computer vision, including semantic segmentation or scene understanding. This paper provides a review on deep learning methods for semantic segmentation applied to various application areas. Firstly, we describe the terminology of this field as well as mandatory background concepts. Next, the main datasets and challenges are exposed to help researchers decide which are the ones that best suit their needs and their targets. Then, existing methods are reviewed, highlighting their contributions and their significance in the field. Finally, quantitative results are given for the described methods and the datasets in which they were evaluated, following up with a discussion of the results. At last, we point out a set of promising future works and draw our own conclusions about the state of the art of semantic segmentation using deep learning techniques.

Motivation & Objective

• Provide a broad survey of datasets useful for segmentation with deep learning techniques.
• Offer an organized review of significant deep learning methods for semantic segmentation and their contributions.
• Summarize performance metrics (accuracy, speed, memory) and compare methods across datasets.
• Discuss challenges and propose directions for future research in semantic segmentation with deep learning.
• Establish context and state of the art to guide researchers entering or advancing in the field.

Proposed method

• Describe the semantic segmentation problem and per-pixel labeling formulation.
• Review common deep network architectures used as building blocks (e.g., AlexNet, VGG, GoogLeNet, ResNet, ReNet).
• Discuss transfer learning and fine-tuning strategies for segmentation networks.
• Explain data preprocessing and augmentation techniques to improve generalization and training efficiency.
• Present and categorize major 2D, 2.5D (RGB-D), and 3D datasets and benchmarks, including their properties and splits.
• Provide a qualitative and quantitative performance evaluation of methods on cited datasets.

Experimental results

Research questions

• RQ1 What datasets and benchmarks are most representative for evaluating deep learning-based semantic segmentation methods?
• RQ2 What architectures and training strategies have proven most effective for segmentation tasks?
• RQ3 How do deep learning approaches compare to traditional methods in terms of accuracy, speed, and memory usage across datasets?
• RQ4 What are the prevailing challenges and future directions in semantic segmentation with deep learning?

Key findings

• The review consolidates a wide range of datasets (2D, 2.5D, and 3D) and clarifies their purposes, classes, formats, and splits.
• Deep learning-based semantic segmentation methods generally outperform traditional approaches and rely on transfer learning and pre-trained networks to handle limited annotated data.
• Transfer learning and fine-tuning are common strategies due to smaller per-pixel labeled datasets for segmentation compared to classification tasks.
• Data augmentation and preprocessing are highlighted as essential for improving generalization, especially on smaller datasets.
• The paper provides performance-oriented discussion and identifies future research directions to advance semantic segmentation with deep learning.