[Paper Review] Deep Clustering for Unsupervised Learning of Visual Features

Mathilde Caron, Piotr Bojanowski+2 more|arXiv (Cornell University)|Jul 15, 2018

Domain Adaptation and Few-Shot LearningComputer Science73 references482 citations

TL;DR

DeepCluster alternates between clustering convnet features with k-means and updating the network to predict cluster assignments, enabling end-to-end unsupervised feature learning that achieves state-of-the-art transfer performance on ImageNet, YFCC100M, and Pascal VOC.

ABSTRACT

Clustering is a class of unsupervised learning methods that has been extensively applied and studied in computer vision. Little work has been done to adapt it to the end-to-end training of visual features on large scale datasets. In this work, we present DeepCluster, a clustering method that jointly learns the parameters of a neural network and the cluster assignments of the resulting features. DeepCluster iteratively groups the features with a standard clustering algorithm, k-means, and uses the subsequent assignments as supervision to update the weights of the network. We apply DeepCluster to the unsupervised training of convolutional neural networks on large datasets like ImageNet and YFCC100M. The resulting model outperforms the current state of the art by a significant margin on all the standard benchmarks.

Motivation & Objective

• Motivate unsupervised learning of universal visual features without manual labels on large-scale datasets.
• Propose a scalable method to learn convnet parameters jointly with cluster assignments.
• Demonstrate that clustering-based pseudo-labels can supervise end-to-end feature learning.
• Show robustness of the approach to architecture choices and data distributions.
• Evaluate transfer performance on standard benchmarks and instance-level retrieval.

Proposed method

• Represent f_theta as a convnet feature extractor and use k-means to assign cluster labels to features.
• Iteratively cluster features to obtain pseudo-labels and update network parameters by predicting these pseudo-labels with a discriminative loss.
• Use PCA reduction and normalization on features before clustering and employ standard SGD with backpropagation for optimization.
• Incorporate simple tricks to avoid trivial solutions such as handling empty clusters and reweighting samples from small clusters.
• Experiment with AlexNet and VGG-16 architectures on ImageNet and YFCC100M; apply Sobel filtering to remove color when needed.
• Evaluate the learned features on downstream tasks including ImageNet/Places classification, Pascal VOC (classification, detection, segmentation), and instance retrieval.

Experimental results

Research questions

• RQ1 Can end-to-end convnet training be effectively driven by clustering-based pseudo-labels without human annotations?
• RQ2 How does iterative clustering of deep features influence the quality and transferability of learned representations?
• RQ3 What architectural choices (e.g., AlexNet vs. VGG-16) and data distributions (ImageNet vs. YFCC100M) affect unsupervised feature learning performance?
• RQ4 How robust is the method to trivial solutions and what strategies prevent collapse (e.g., empty clusters, imbalanced pseudo-labels)?

Key findings

• DeepCluster achieves state-of-the-art performance among unsupervised feature learning methods on standard transfer tasks after pretraining on large-scale datasets.
• Using deeper architectures (e.g., VGG-16) with DeepCluster yields substantial gains over AlexNet in transfer tasks.
• Pretraining on uncured data distributions (YFCC100M) still yields strong transfer performance, illustrating robustness to data biases.
• Linear probes on frozen layers show that higher-layer features trained with DeepCluster are competitive with supervised counterparts on several tasks.
• DeepCluster offers notable gains on Pascal VOC, particularly in semantic segmentation, and remains effective for instance retrieval, underscoring the value of instance-level information in learned features.