[Paper Review] SegDiscover: Visual Concept Discovery via Unsupervised Semantic Segmentation

Haiyang Huang, Zhi Chen+1 more|arXiv (Cornell University)|Apr 22, 2022

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

SegDiscover proposes an unsupervised framework for visual concept discovery by reframing the task as unsupervised semantic segmentation, leveraging self-supervised feature learning and neural network refinement to discover coherent, multi-concept segments in complex scenes without human annotations. It achieves state-of-the-art performance on Cityscapes and COCO-Stuff, outperforming prior unsupervised methods and revealing interpretable, concept-based explanations via different pretrained encoders.

ABSTRACT

Visual concept discovery has long been deemed important to improve interpretability of neural networks, because a bank of semantically meaningful concepts would provide us with a starting point for building machine learning models that exhibit intelligible reasoning process. Previous methods have disadvantages: either they rely on labelled support sets that incorporate human biases for objects that are "useful," or they fail to identify multiple concepts that occur within a single image. We reframe the concept discovery task as an unsupervised semantic segmentation problem, and present SegDiscover, a novel framework that discovers semantically meaningful visual concepts from imagery datasets with complex scenes without supervision. Our method contains three important pieces: generating concept primitives from raw images, discovering concepts by clustering in the latent space of a self-supervised pretrained encoder, and concept refinement via neural network smoothing. Experimental results provide evidence that our method can discover multiple concepts within a single image and outperforms state-of-the-art unsupervised methods on complex datasets such as Cityscapes and COCO-Stuff. Our method can be further used as a neural network explanation tool by comparing results obtained by different encoders.

Motivation & Objective

• To address the challenge of discovering semantically meaningful visual concepts in complex natural images without human-annotated supervision.
• To overcome biases in prior methods that rely on labeled support sets or focus only on salient foreground objects.
• To enable discovery of multiple, coherent concepts within a single image, including background and contextual elements.
• To provide a foundation for interpretable machine learning by generating concept-based explanations from neural network activations.
• To demonstrate that self-supervised encoders can yield better concept discovery than supervised ones due to reduced annotation bias.

Proposed method

• Generating concept primitives by applying unsupervised image segmentation to produce patch- or superpixel-level regions from raw images.
• Discovering visual concepts by clustering these primitives in the latent space of a self-supervised, pretrained convolutional encoder.
• Refining discovered concepts through a neural network training process that maps cluster labels back to images and learns to predict pseudo-segmentation labels.
• Utilizing multiple self-supervised encoders (MoCo, SwAV, DeepCluster-V2) and a supervised ImageNet encoder to compare concept discovery performance and interpretability.
• Applying overclustering with K=100 to improve segmentation quality and reduce overfitting in the latent space clustering step.
• Validating results via mIoU, wIoU, and pAcc on Cityscapes and COCO-Stuff, with qualitative analysis of concept maps across different encoders.

Experimental results

Research questions

• RQ1 Can visual concepts be discovered in complex, real-world scenes without any human-annotated supervision?
• RQ2 Can a framework simultaneously identify multiple, distinct concepts within a single image, including background and contextual elements?
• RQ3 How does the choice of pretrained encoder (self-supervised vs. supervised) affect the quality and interpretability of discovered visual concepts?
• RQ4 Can the discovered concepts serve as a basis for explaining neural network behavior through concept-based saliency maps?
• RQ5 Does incorporating global contextual information during refinement improve the coherence and accuracy of discovered concepts?

Key findings

• SegDiscover achieves state-of-the-art unsupervised semantic segmentation performance on Cityscapes and COCO-Stuff, with mIoU of 13.62 and 14.34 using SwAV and DeepCluster-V2 encoders, respectively.
• Using self-supervised encoders (especially DeepCluster-V2) yields better performance than using a supervised ImageNet classifier, with a 4+ point improvement in pAcc.
• The method successfully discovers multiple, coherent visual concepts within a single image, including non-salient and background elements such as walls, ceilings, and distant plants.
• SegDiscover with DeepCluster-V2 outperforms the current SOTA method (PiCIE) on COCO-Stuff, demonstrating superior generalization and concept coherence.
• Qualitative analysis shows that different encoders lead to distinct concept maps—e.g., MoCo tends to produce smaller segments, while SwAV sometimes misclassifies parts of buildings as 'ground'—indicating encoder-dependent inductive biases.
• The framework enables concept-based explanation of neural networks: different encoders highlight different regions of interest, revealing structural differences in their learned representations.