[Paper Review] A Survey on Label-efficient Deep Image Segmentation: Bridging the Gap between Weak Supervision and Dense Prediction

Wei Shen, Zelin Peng+7 more|arXiv (Cornell University)|Jul 4, 2022

Advanced Neural Network Applications4 citations

TL;DR

This survey presents a comprehensive taxonomy of label-efficient deep image segmentation methods that bridge weak supervision (e.g., bounding boxes, scribbles, image-level labels) with dense pixel-level prediction. It systematizes existing approaches by supervision type and segmentation task, identifies key inductive biases like cross-pixel similarity and cross-view consistency, and outlines future directions including vision transformers and zero-shot segmentation with text supervision.

ABSTRACT

The rapid development of deep learning has made a great progress in image segmentation, one of the fundamental tasks of computer vision. However, the current segmentation algorithms mostly rely on the availability of pixel-level annotations, which are often expensive, tedious, and laborious. To alleviate this burden, the past years have witnessed an increasing attention in building label-efficient, deep-learning-based image segmentation algorithms. This paper offers a comprehensive review on label-efficient image segmentation methods. To this end, we first develop a taxonomy to organize these methods according to the supervision provided by different types of weak labels (including no supervision, inexact supervision, incomplete supervision and inaccurate supervision) and supplemented by the types of segmentation problems (including semantic segmentation, instance segmentation and panoptic segmentation). Next, we summarize the existing label-efficient image segmentation methods from a unified perspective that discusses an important question: how to bridge the gap between weak supervision and dense prediction -- the current methods are mostly based on heuristic priors, such as cross-pixel similarity, cross-label constraint, cross-view consistency, and cross-image relation. Finally, we share our opinions about the future research directions for label-efficient deep image segmentation.

Motivation & Objective

• To address the high cost and labor intensity of pixel-level annotations in deep image segmentation by enabling label-efficient learning.
• To systematize the rapidly growing body of research on weakly supervised, semi-supervised, and self-supervised image segmentation methods.
• To unify diverse label-efficient methods under a common framework that bridges weak supervision and dense prediction.
• To identify key inductive biases—such as cross-pixel similarity, cross-view consistency, and cross-image relations—used across state-of-the-art approaches.
• To outline future research directions, including zero-shot segmentation with text supervision, vision transformers, and lifelong learning with mixed annotations.

Proposed method

• Proposes a hierarchical taxonomy of weak supervision types: no supervision, inexact, incomplete, and inaccurate supervision.
• Classifies methods according to segmentation task: semantic, instance, and panoptic segmentation.
• Identifies and analyzes core inductive biases used in label-efficient models: cross-pixel similarity, cross-label constraints, cross-view consistency, and cross-image relations.
• Reviews representative methods under each supervision type, such as MaskContrast, STEGO, PseudoSeg, and ADELE, to illustrate design principles.
• Analyzes how self-supervised pre-training (e.g., DINO, MAE, BEiT) enables pseudo-label generation for weakly supervised segmentation.
• Proposes a unified perspective on how to close the gap between weak supervision and dense prediction through structured inductive biases.

Experimental results

Research questions

• RQ1 How can weak supervision types (e.g., image-level labels, bounding boxes, scribbles) be systematically categorized to guide method design in label-efficient segmentation?
• RQ2 What are the dominant inductive biases that enable effective knowledge distillation from weak supervision to dense pixel-level predictions?
• RQ3 How do cross-pixel similarity, cross-view consistency, and cross-image relations contribute to improving segmentation accuracy under weak supervision?
• RQ4 What are the key limitations of current label-efficient segmentation methods, and how can they be addressed through new architectures or pre-training paradigms?
• RQ5 What future research directions—such as vision transformers, zero-shot segmentation, or lifelong learning—hold the most promise for advancing label efficiency in dense prediction tasks?

Key findings

• The use of self-supervised vision transformers (e.g., ViT with DINO, MAE, or iBoT pre-training) generates high-quality attention maps that can serve as reliable pseudo-labels for weakly supervised segmentation.
• Methods leveraging cross-pixel similarity and cross-view consistency, such as STEGO and PseudoSeg, achieve strong performance on weakly supervised semantic segmentation benchmarks with minimal supervision.
• Vision transformer-based models like MCTFormer and CLIMS show improved performance in weakly supervised semantic segmentation by modeling long-range dependencies more effectively than CNNs.
• Zero-shot segmentation using CLIP-style text-image alignment is a promising direction, though current methods still rely on pre-training on similar classes, limiting true zero-shot generalization.
• Unexplored problems such as instance segmentation from noisy labels and panoptic segmentation from incomplete labels remain open, indicating significant research potential.
• Lifelong learning with mixed annotation types (e.g., bounding boxes, scribbles, image-level labels) is a critical future direction, requiring new benchmarks and unified frameworks for evaluation and integration.