[Paper Review] Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images

Yan Yang, Bo Yang|arXiv (Cornell University)|Oct 5, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper investigates why existing unsupervised object segmentation models fail on real-world images despite strong performance on synthetic datasets. By introducing four quantitative complexity factors measuring appearance and geometry biases, the authors demonstrate that the mismatch between model inductive biases and real-world objectness distributions is the core failure cause, urging future work to embed explicit objectness priors in network design.

ABSTRACT

In this paper, we study the problem of unsupervised object segmentation from single images. We do not introduce a new algorithm, but systematically investigate the effectiveness of existing unsupervised models on challenging real-world images. We firstly introduce four complexity factors to quantitatively measure the distributions of object- and scene-level biases in appearance and geometry for datasets with human annotations. With the aid of these factors, we empirically find that, not surprisingly, existing unsupervised models catastrophically fail to segment generic objects in real-world images, although they can easily achieve excellent performance on numerous simple synthetic datasets, due to the vast gap in objectness biases between synthetic and real images. By conducting extensive experiments on multiple groups of ablated real-world datasets, we ultimately find that the key factors underlying the colossal failure of existing unsupervised models on real-world images are the challenging distributions of object- and scene-level biases in appearance and geometry. Because of this, the inductive biases introduced in existing unsupervised models can hardly capture the diverse object distributions. Our research results suggest that future work should exploit more explicit objectness biases in the network design.

Motivation & Objective

• To investigate the true potential of unsupervised object segmentation models on real-world single images, challenging the assumption that strong synthetic performance implies real-world applicability.
• To identify the root causes behind the failure of state-of-the-art unsupervised models when applied to complex real-world scenes like COCO.
• To quantify the difficulty of object and scene segmentation using appearance and geometric biases, moving beyond qualitative Gestalt principles.
• To evaluate the robustness of four leading unsupervised models (AIR, MONet, IODINE, SlotAtt) across ablated real-world datasets with controlled bias variations.
• To guide future research by identifying that current inductive biases in models are fundamentally mismatched with real-world objectness distributions.

Proposed method

• Propose four quantitative complexity factors—object color gradient, inter-object color similarity, object shape regularity, and scale uniformity—to measure appearance and geometry biases at both object and scene levels.
• Curate six real-world datasets (YCB, ScanNet, COCO) with systematic ablations of these four factors to isolate their impact on segmentation performance.
• Train 130+ models from scratch using four representative unsupervised methods (AIR, MONet, IODINE, SlotAtt) on ablated datasets to assess sensitivity to bias distributions.
• Use standard metrics (AP, PQ, Precision, Recall) to quantitatively evaluate segmentation performance across all ablated settings and model variants.
• Analyze qualitative results across multiple datasets to visualize failure modes and success patterns under different bias conditions.
• Compare performance across synthetic (e.g., dSprites, CLEVR) and real-world datasets to highlight the generalization gap caused by bias distribution shifts.

Experimental results

Research questions

• RQ1 To what extent do existing unsupervised object segmentation models generalize to real-world single images, given their strong performance on synthetic benchmarks?
• RQ2 Which specific appearance and geometric factors in real-world images most significantly degrade unsupervised segmentation performance?
• RQ3 How do ablations of object- and scene-level biases (e.g., color gradient, shape regularity) affect the segmentation accuracy of unsupervised models?
• RQ4 Why do current unsupervised models fail to discover generic objects in real-world images despite learning powerful inductive biases?
• RQ5 To what extent are the inductive biases in existing models mismatched with the actual objectness distributions found in real-world images?

Key findings

• Existing unsupervised object segmentation models catastrophically fail on real-world images, achieving only 29.4% AP on COCO-S+T+U, compared to over 80% on synthetic datasets.
• The primary cause of failure is the mismatch between the inductive biases in current models and the complex, diverse distributions of appearance and geometry biases in real-world images.
• Models show high sensitivity to appearance factors: performance drops significantly when object color gradient and inter-object color similarity are preserved, especially in COCO and ScanNet.
• Segmentation accuracy improves substantially when appearance factors are ablated (e.g., on C+T datasets), indicating that appearance complexity is a dominant failure factor.
• Even with ablated appearance factors, models still struggle on real-world data, highlighting the persistent challenge of geometric biases such as irregular shapes and non-uniform scales.
• MONet achieves 86.9% AP on COCO-C+T+U, but only 29.4% on COCO-S+T+U, demonstrating that the presence of appearance variation severely degrades performance despite strong inductive biases.