[Paper Review] Joint Unsupervised Learning of Deep Representations and Image Clusters

Jianwei Yang, Devi Parikh+1 more|arXiv (Cornell University)|Apr 13, 2016

Domain Adaptation and Few-Shot Learning47 references145 citations

TL;DR

A recurrent framework that jointly learns deep CNN representations and image clusters in an unsupervised manner by alternating clustering in the forward pass and representation learning in the backward pass, optimized with a weighted triplet loss.

ABSTRACT

In this paper, we propose a recurrent framework for Joint Unsupervised LEarning (JULE) of deep representations and image clusters. In our framework, successive operations in a clustering algorithm are expressed as steps in a recurrent process, stacked on top of representations output by a Convolutional Neural Network (CNN). During training, image clusters and representations are updated jointly: image clustering is conducted in the forward pass, while representation learning in the backward pass. Our key idea behind this framework is that good representations are beneficial to image clustering and clustering results provide supervisory signals to representation learning. By integrating two processes into a single model with a unified weighted triplet loss and optimizing it end-to-end, we can obtain not only more powerful representations, but also more precise image clusters. Extensive experiments show that our method outperforms the state-of-the-art on image clustering across a variety of image datasets. Moreover, the learned representations generalize well when transferred to other tasks.

Motivation & Objective

• Motivate unsupervised learning of representations without labels through joint clustering.
• Develop a recurrent framework that unifies agglomerative clustering with deep representation learning.
• Derive a single loss to guide both clustering and representation learning in an end-to-end fashion.
• Demonstrate superior clustering performance and transferable representations across datasets.

Proposed method

• Represent images with a CNN to obtain deep features for clustering.
• Formulate agglomerative clustering as a forward-pass recurrent process merging clusters over timesteps.
• Define a directed affinity graph and use graph degree linkage to measure cluster affinity.
• Introduce a weighted triplet loss that combines cluster affinities and local structure to guide learning.
• Train in partially unrolled periods, alternating forward (clustering) and backward (representation learning) passes.
• Optimize a total loss that sums period-wise objectives and enables end-to-end learning.

Experimental results

Research questions

• RQ1 Can joint unsupervised learning of deep representations and image clusters outperform fixed-representation clustering methods?
• RQ2 Do learned representations transfer across datasets and clustering algorithms while preserving or improving clustering quality?
• RQ3 How does incorporating local cluster structure into the clustering loss affect optimization and results?
• RQ4 Can the learned representations improve performance on downstream tasks such as face verification and image classification without labels?
• RQ5 Is agglomerative clustering suitable for integration into a recurrent deep learning framework?

Key findings

• The joint approach outperforms state-of-the-art clustering methods on multiple datasets using NMI as the metric.
• Learned representations transfer across clustering algorithms and improve their performance on various datasets.
• On COIL20 and CMU-PIE, the method achieves perfect NMI (1.0) in the reported results.
• The learned representations enable competitive or superior results for face verification on LFW under an unsupervised training setup.
• Transfer experiments show cross-dataset clustering improvements when using representations from one dataset on another.
• The framework yields transferable representations that also support image classification pipelines with unsupervised learning.