[Paper Review] Unsupervised Semantic Segmentation with Self-supervised Object-centric Representations

Andrii Zadaianchuk, Matthaeus Kleindessner+3 more|arXiv (Cornell University)|Jul 11, 2022

Advanced Image and Video Retrieval Techniques16 citations

TL;DR

This paper proposes COMUS, a self-supervised method for unsupervised semantic segmentation that leverages object-centric representations from DINO and unsupervised saliency detection to generate pseudo-masks for training a segmentation network. It achieves state-of-the-art performance on PASCAL VOC (50.0 mIoU) and reports the first unsupervised results on the full 81-class MS COCO dataset, attaining 19.6 mIoU average and discovering 34 categories with >20% IoU.

ABSTRACT

In this paper, we show that recent advances in self-supervised feature learning enable unsupervised object discovery and semantic segmentation with a performance that matches the state of the field on supervised semantic segmentation 10 years ago. We propose a methodology based on unsupervised saliency masks and self-supervised feature clustering to kickstart object discovery followed by training a semantic segmentation network on pseudo-labels to bootstrap the system on images with multiple objects. We present results on PASCAL VOC that go far beyond the current state of the art (50.0 mIoU), and we report for the first time results on MS COCO for the whole set of 81 classes: our method discovers 34 categories with more than $20\%$ IoU, while obtaining an average IoU of 19.6 for all 81 categories.

Motivation & Objective

• To develop a simple yet strong baseline for unsupervised semantic segmentation without human-annotated categories.
• To enable unsupervised discovery of object categories and precise localization in multi-object images using self-supervised representations.
• To extend unsupervised semantic segmentation to large-scale datasets like MS COCO with all 81 classes, previously unexplored in this setting.
• To demonstrate that unsupervised methods can reach performance levels comparable to supervised methods from 10 years ago.
• To identify key components and bottlenecks in the pipeline through ablation studies for future improvement.

Proposed method

• Utilizes DINO, a self-supervised vision transformer, to extract high-quality, object-centric features from ImageNet without labels.
• Applies unsupervised saliency detectors (DeepUSPS or BASNet) to localize foreground objects and extract region proposals.
• Applies spectral clustering on DINO features within saliency regions to discover object categories and generate pseudo-masks.
• Filters uncertain clusters to improve mask quality and reduce noise in pseudo-labels.
• Trains a DeepLabv3 semantic segmentation network on the generated pseudo-masks using self-training with data augmentation.
• Performs iterative self-training to refine predictions and improve segmentation quality on both PASCAL VOC and MS COCO.

Experimental results

Research questions

• RQ1 Can self-supervised object-centric representations enable strong unsupervised semantic segmentation without any human annotations?
• RQ2 How effective is the combination of unsupervised saliency detection and feature clustering for object discovery in complex, multi-object scenes?
• RQ3 To what extent can a self-training loop on pseudo-masks improve segmentation performance on datasets like MS COCO with 81 classes?
• RQ4 What are the key components of the pipeline that most influence performance, and where are the main bottlenecks for improvement?
• RQ5 Can unsupervised segmentation methods achieve performance levels comparable to supervised methods from 10 years ago?

Key findings

• COMUS achieves 50.0 mIoU on PASCAL VOC, matching the performance of supervised semantic segmentation methods from 10 years ago.
• For the first time, the paper reports unsupervised semantic segmentation results on the full 81-class MS COCO dataset, with an average mIoU of 19.6.
• The method successfully discovers 34 object categories on MS COCO with more than 20% IoU, demonstrating effective multi-object and multi-category discovery.
• Ablation studies show that unsupervised saliency detection and spectral clustering on DINO features are critical for initial mask quality and category discovery.
• Self-training significantly improves performance, with two iterations on PASCAL VOC and one on MS COCO yielding consistent gains.
• The computational cost is dominated by DINO pretraining (3 days on 8 GPUs), but inference and downstream training are efficient, with full self-training taking ~1 hour on 4 T4 GPUs.