[Paper Review] Invariant Information Distillation for Unsupervised Image Segmentation and Clustering

Xu Ji, João F. Henriques+1 more|arXiv (Cornell University)|Jul 17, 2018

Advanced Image and Video Retrieval TechniquesComputer Science17 references53 citations

TL;DR

This paper proposes a novel unsupervised clustering method that trains a neural network from scratch using only unpaired image data by maximizing mutual information between predictions on augmented image pairs. It achieves state-of-the-art performance on eight benchmarks, including 9.5% higher accuracy than prior methods on CIFAR10 and 88.8% accuracy on STL10 in a semi-supervised setting.

ABSTRACT

We present a novel clustering objective that learns a neural network classifier from scratch, given only unlabelled data samples. The model discovers clusters that accurately match semantic classes, achieving state-of-the-art results in eight unsupervised clustering benchmarks spanning image classification and segmentation. These include STL10, an unsupervised variant of ImageNet, and CIFAR10, where we significantly beat the accuracy of our closest competitors by 6.6 and 9.5 absolute percentage points respectively. The method is not specialised to computer vision and operates on any paired dataset samples; in our experiments we use random transforms to obtain a pair from each image. The trained network directly outputs semantic labels, rather than high dimensional representations that need external processing to be usable for semantic clustering. The objective is simply to maximise mutual information between the class assignments of each pair. It is easy to implement and rigorously grounded in information theory, meaning we effortlessly avoid degenerate solutions that other clustering methods are susceptible to. In addition to the fully unsupervised mode, we also test two semi-supervised settings. The first achieves 88.8% accuracy on STL10 classification, setting a new global state-of-the-art over all existing methods (whether supervised, semi-supervised or unsupervised). The second shows robustness to 90% reductions in label coverage, of relevance to applications that wish to make use of small amounts of labels. this http URL

Motivation & Objective

• To develop a clustering objective that learns semantic representations from unlabelled data without requiring pre-trained features or external clustering steps.
• To address the issue of degenerate solutions common in unsupervised contrastive learning by grounding the method in information theory.
• To enable direct prediction of semantic labels via the network, avoiding high-dimensional representation spaces that require post-processing.
• To evaluate performance across diverse benchmarks including image classification, segmentation, and semi-supervised learning settings.
• To demonstrate robustness under extreme label scarcity, such as 90% label reduction, in semi-supervised scenarios.

Proposed method

• The method uses random data augmentations to create paired samples from each image, forming positive pairs for contrastive learning.
• It maximizes mutual information between the network's predictions on each augmented pair, using an invariant information distillation objective.
• The network is trained to output semantic class assignments directly, bypassing the need for downstream clustering algorithms.
• The objective is grounded in information theory, ensuring robustness against degenerate solutions common in other contrastive methods.
• The approach is generalizable beyond vision, applicable to any paired data distribution.
• Semi-supervised variants are tested by combining a small number of labels with the unsupervised objective, improving generalization under label scarcity.

Experimental results

Research questions

• RQ1 Can a neural network be trained from scratch to discover meaningful semantic clusters using only unlabelled data?
• RQ2 Does maximizing mutual information between predictions on augmented image pairs lead to state-of-the-art clustering performance?
• RQ3 Can the method generalize across diverse benchmarks, including image classification and segmentation tasks?
• RQ4 How does the method perform under extreme label scarcity, such as 90% label reduction?
• RQ5 Can the model achieve SOTA results in semi-supervised learning without relying on large-scale labelled data?

Key findings

• The method achieves a 9.5% absolute accuracy gain over the closest competitor on CIFAR10 in unsupervised clustering.
• On STL10, the semi-supervised variant achieves 88.8% accuracy, setting a new global SOTA across all supervised, semi-supervised, and unsupervised methods.
• The model significantly outperforms prior approaches on eight unsupervised benchmarks, including STL10, ImageNet-unsupervised, and CIFAR10.
• The method is robust to 90% reduction in label coverage, demonstrating strong performance with minimal supervision.
• The approach avoids degenerate solutions due to its information-theoretic foundation, ensuring stable and meaningful clustering.
• The network directly outputs semantic labels, eliminating the need for post-processing or external clustering steps.