[Paper Review] SELF-LABELLING VIA SIMULTANEOUS CLUSTERING AND REPRESENTATION LEARNING

Yuki M. Asano, Rupprecht, Christian+1 more|arXiv (Cornell University)|Nov 13, 2019

Advanced Image and Video Retrieval TechniquesComputer Science45 references98 citations

TL;DR

The paper introduces SeLa, a principled self-labelling method that jointly learns representations and pseudolabels by casting labeling as an optimal transport problem with equipartition, solved efficiently via a Sinkhorn-based algorithm. It yields state-of-the-art self-supervised representations on ImageNet, CIFAR, SVHN, and Pascal VOC baselines.

ABSTRACT

Combining clustering and representation learning is one of the most promising approaches for unsupervised learning of deep neural networks. However, doing so naively leads to ill posed learning problems with degenerate solutions. In this paper, we propose a novel and principled learning formulation that addresses these issues. The method is obtained by maximizing the information between labels and input data indices. We show that this criterion extends standard crossentropy minimization to an optimal transport problem, which we solve efficiently for millions of input images and thousands of labels using a fast variant of the Sinkhorn-Knopp algorithm. The resulting method is able to self-label visual data so as to train highly competitive image representations without manual labels. Our method achieves state of the art representation learning performance for AlexNet and ResNet-50 on SVHN, CIFAR-10, CIFAR-100 and ImageNet and yields the first self-supervised AlexNet that outperforms the supervised Pascal VOC detection baseline. Code and models are available.

Motivation & Objective

• Motivate unsupervised learning of deep representations without manual labels.
• Propose a principled self-labelling framework that avoids degenerate solutions.
• Couple representation learning with clustering via a single cross-entropy objective under equipartition constraints.
• Scale the approach to millions of images and thousands of labels with an efficient Sinkhorn-based solver.

Proposed method

• Formulate labeling as a constrained cross-entropy objective, introducing an equipartition constraint to avoid degenerate solutions.
• Show that the label assignment problem is an optimal transport problem over a transportation polytope.
• Relax and solve the transport problem with a fast Sinkhorn-Knopp variant using KL regularization for scalability.
• Alternate between representation learning (minimizing cross-entropy with current labels) and self-labelling (updating pseudo-labels via Sinkhorn updates).
• Explain the interpretation as maximizing mutual information between data indices and labels under equipartition.

Experimental results

Research questions

• RQ1 Can a unified objective for simultaneous clustering and representation learning be derived that guarantees convergence and avoids degenerate solutions?
• RQ2 Does enforcing equipartition via an optimal-transport formulation improve self-supervised learning performance over naive clustering-plus-ce methods?
• RQ3 How scalable is the approach to large datasets (millions of images) and thousands of labels?
• RQ4 What are the benefits of using multiple clustering heads and data augmentations in stabilizing and improving representations?

Key findings

• Achieves state-of-the-art representation learning performance for AlexNet and ResNet-50 on SVHN, CIFAR-10, CIFAR-100 and ImageNet.
• Outperforms other self-supervised methods that rely on a single supervisory signal and is competitive with recent contrastive approaches.
• First self-supervised AlexNet to outperform the supervised Pascal VOC detection baseline.
• Label transfer experiments show the learned pseudo-labels enable competitive or superior performance when training from scratch or transferring between architectures.
• Demonstrates robustness to data imbalance with equipartition regularization acting as a regularizer against degenerate solutions.