[Paper Review] A probabilistic constrained clustering for transfer learning and image category discovery

Yen-Chang Hsu, Zhaoyang Lv+3 more|arXiv (Cornell University)|Jun 28, 2018

Domain Adaptation and Few-Shot Learning16 references3 citations

TL;DR

This paper proposes a novel probabilistic constrained clustering (CCL) objective that directly minimizes the negative log-likelihood of cluster assignments under pairwise similarity and dissimilarity constraints, enabling end-to-end joint optimization of deep feature representations and cluster assignments. CCL achieves superior scalability and performance over prior methods—especially on deeper networks and larger numbers of clusters—demonstrating state-of-the-art results in transfer learning and image category discovery with no hyperparameters and improved robustness to noisy constraints.

ABSTRACT

Neural network-based clustering has recently gained popularity, and in particular a constrained clustering formulation has been proposed to perform transfer learning and image category discovery using deep learning. The core idea is to formulate a clustering objective with pairwise constraints that can be used to train a deep clustering network; therefore the cluster assignments and their underlying feature representations are jointly optimized end-to-end. In this work, we provide a novel clustering formulation to address scalability issues of previous work in terms of optimizing deeper networks and larger amounts of categories. The proposed objective directly minimizes the negative log-likelihood of cluster assignment with respect to the pairwise constraints, has no hyper-parameters, and demonstrates improved scalability and performance on both supervised learning and unsupervised transfer learning.

Motivation & Objective

• To address scalability limitations in existing constrained clustering methods when applied to deeper neural networks and larger numbers of clusters.
• To develop a clustering objective that jointly optimizes feature representations and cluster assignments end-to-end using pairwise constraints.
• To improve performance in unsupervised image category discovery and transfer learning by leveraging a likelihood-based formulation over contrastive loss.
• To eliminate hyperparameter dependence by using a principled probabilistic likelihood objective instead of KL-divergence-based losses.
• To evaluate robustness in real-world settings with noisy, automatically generated constraints.

Proposed method

• Formulates cluster assignment probabilities via a neural network outputting a categorical distribution over k clusters using softmax.
• Defines the probability of two samples being assigned to the same cluster as the dot product of their predicted cluster distributions: P(y_i = y_j) = f_θ(x_i)^T f_θ(x_j).
• Models pairwise constraints as similar (must-link) or dissimilar (cannot-link) pairs, with P(y_i = y_j) for similar pairs and P(y_i ≠ y_j) = 1 - P(y_i = y_j) for dissimilar pairs.
• Constructs a likelihood objective L(θ|S⁺,S⁻) as the product of probabilities over all similar and dissimilar pairs, which is minimized via negative log-likelihood during training.
• Uses stochastic gradient descent to optimize the network parameters θ, with final cluster assignments obtained by forward-passing data through the trained network.
• Employs a joint optimization framework where feature learning and clustering are trained end-to-end using only pairwise constraints as supervision.

Experimental results

Research questions

• RQ1 Can a likelihood-based clustering objective outperform KL-divergence-based methods in terms of scalability on deeper networks?
• RQ2 How does the proposed method perform when the number of clusters exceeds 10, especially in unsupervised category discovery?
• RQ3 Does the absence of hyperparameters improve robustness and performance in real-world, noisy constraint settings?
• RQ4 Can the method achieve state-of-the-art performance in transfer learning for image category discovery with automatically generated constraints?
• RQ5 How does the method scale to large-scale datasets like ImageNet with 100 or more categories?

Key findings

• On CIFAR-100 with VGG8, CCL achieves 64.0% clustering accuracy, significantly outperforming KCL’s 44.3% under the same setup.
• On ResNet-18 features from ImageNet, CCL achieves 74.4% clustering accuracy for 30 held-out categories, surpassing CCN-KCL’s 73.8%.
• In the unknown cluster number setting (k=100), CCL achieves 71.5% accuracy, a substantial improvement over KCL’s 65.2% on ImageNet.
• On Omniglot, CCL achieves 83.3% accuracy with ground-truth cluster count and 80.2% with k=100, outperforming CCN-KCL (82.4% and 78.1% respectively).
• CCL maintains high performance even on very deep networks like PReActResNet-101, achieving 93.3% accuracy compared to KCL’s 21.9% on CIFAR-10.
• The method demonstrates consistent superiority in normalized mutual information (NMI), with CCL achieving 0.897 NMI on Omniglot (vs. 0.889 for KCL), indicating better cluster quality.