[Paper Review] Deep Clustering via Joint Convolutional Autoencoder Embedding and Relative Entropy Minimization

Kamran Ghasedi Dizaji, Amirhossein Herandi+3 more|arXiv (Cornell University)|Apr 20, 2017

Domain Adaptation and Few-Shot Learning56 references57 citations

TL;DR

DEPICT jointly learns a deep embedding and clustering by minimizing KL divergence with a cluster-frequency prior, using end-to-end training with reconstruction loss for regularization, achieving strong clustering performance and faster runtimes without labeled hyper-parameter tuning.

ABSTRACT

Image clustering is one of the most important computer vision applications, which has been extensively studied in literature. However, current clustering methods mostly suffer from lack of efficiency and scalability when dealing with large-scale and high-dimensional data. In this paper, we propose a new clustering model, called DEeP Embedded RegularIzed ClusTering (DEPICT), which efficiently maps data into a discriminative embedding subspace and precisely predicts cluster assignments. DEPICT generally consists of a multinomial logistic regression function stacked on top of a multi-layer convolutional autoencoder. We define a clustering objective function using relative entropy (KL divergence) minimization, regularized by a prior for the frequency of cluster assignments. An alternating strategy is then derived to optimize the objective by updating parameters and estimating cluster assignments. Furthermore, we employ the reconstruction loss functions in our autoencoder, as a data-dependent regularization term, to prevent the deep embedding function from overfitting. In order to benefit from end-to-end optimization and eliminate the necessity for layer-wise pretraining, we introduce a joint learning framework to minimize the unified clustering and reconstruction loss functions together and train all network layers simultaneously. Experimental results indicate the superiority and faster running time of DEPICT in real-world clustering tasks, where no labeled data is available for hyper-parameter tuning.

Motivation & Objective

• Address inefficiency and scalability of clustering in high-dimensional, large-scale image data.
• Develop a deep, discriminative embedding subspace suitable for accurate cluster assignments.
• Eliminate layer-wise pretraining by introducing a unified end-to-end optimization combining clustering and reconstruction losses.
• Mitigate overfitting through autoencoder-based reconstruction losses as data-dependent regularization.
• Demonstrate competitive or superior clustering performance with reduced need for labeled data tuning.

Proposed method

• Proposes DEPICT, a model with a multinomial logistic regression (soft-max) layer atop a multi-layer convolutional autoencoder.
• Defines a clustering objective via KL divergence minimization between model predictions and an auxiliary target, plus a prior on cluster frequencies (uniform prior by default).
• Uses an alternating optimization: update target Q (E-step) with a closed-form approximation, then update network parameters (M-step) via backpropagation with cross-entropy on Q.
• Implements a joint learning framework that trains all encoder/decoder layers simultaneously, incorporating reconstruction losses as data-dependent regularization.
• Extends to a denoising autoencoder setting (MdA) with a corrupted encoder and a clean encoder; clustering is performed on the clean embedding while predictions come from the corrupted path, enforcing noise-invariant clustering.
• Trains end-to-end without layer-wise pretraining, via a unified objective that combines clustering loss and reconstruction losses across all layers.

Experimental results

Research questions

• RQ1 Can a joint deep embedding and clustering model improve clustering performance on high-dimensional image data without labeled hyper-parameter tuning?
• RQ2 Does end-to-end joint optimization with reconstruction-based regularization yield more discriminative embeddings than non-joint or layer-wise pretraining approaches?
• RQ3 How does incorporating a cluster-frequency prior affect cluster balance and outlier avoidance in deep clustering?
• RQ4 What are the benefits of using a denoising autoencoder framework (MdA) for noise-robust clustering in an unsupervised setting?
• RQ5 How does DEPICT compare to state-of-the-art clustering methods in terms of accuracy and NMI on standard image datasets, and what about running time?

Key findings

• DEPICT achieves competitive or superior clustering performance on benchmark image datasets without supervised hyper-parameter tuning.
• Joint end-to-end training (MdA) yields a more discriminative embedding subspace than non-joint or layer-wise pretraining approaches.
• Incorporating a KL-divergence clustering objective with a cluster-frequency prior improves cluster balance and reduces outlier assignment.
• Reconstruction losses as data-dependent regularization help prevent overfitting and improve generalization in clustering.
• DEPICT demonstrates faster running times compared to several state-of-the-art deep clustering methods, especially on large-scale datasets.