[Paper Review] Meta-Unsupervised-Learning: A supervised approach to unsupervised learning

Vikas Garg, Adam Tauman Kalai|arXiv (Cornell University)|Dec 29, 2016

Anomaly Detection Techniques and Applications14 references3 citations

TL;DR

This paper introduces meta-unsupervised-learning, a framework that reduces unsupervised learning tasks—such as clustering, outlier detection, and similarity prediction—to supervised learning by leveraging prior labeled datasets. By training on a diverse repository of classification problems, the method learns to select optimal clustering algorithms, hyperparameters, and outlier thresholds, significantly outperforming baseline heuristics even on unseen data, and circumvents Kleinberg’s impossibility theorem for clustering.

ABSTRACT

We introduce a new paradigm to investigate unsupervised learning, reducing unsupervised learning to supervised learning. Specifically, we mitigate the subjectivity in unsupervised decision-making by leveraging knowledge acquired from prior, possibly heterogeneous, supervised learning tasks. We demonstrate the versatility of our framework via comprehensive expositions and detailed experiments on several unsupervised problems such as (a) clustering, (b) outlier detection, and (c) similarity prediction under a common umbrella of meta-unsupervised-learning. We also provide rigorous PAC-agnostic bounds to establish the theoretical foundations of our framework, and show that our framing of meta-clustering circumvents Kleinberg's impossibility theorem for clustering.

Motivation & Objective

• To address the inherent subjectivity and lack of objective evaluation in unsupervised learning by framing it as a meta-learning problem.
• To reduce the difficulty of selecting clustering algorithms, number of clusters, and outlier thresholds through data-driven, transferable knowledge from prior supervised tasks.
• To provide a theoretically grounded approach that circumvents Kleinberg’s impossibility theorem for clustering by leveraging a meta-distribution over unsupervised problems.
• To demonstrate that heterogeneous, prior labeled datasets can be used to improve performance on new, unlabeled unsupervised problems.
• To show that neural networks trained on aggregated, unlabeled data from diverse domains can generalize to new unsupervised tasks.

Proposed method

• Treat each labeled dataset as a meta-example, where the ground-truth clustering (from labels) defines the target for unsupervised learning.
• Reduce unsupervised problems like clustering and similarity prediction to pairwise binary classification, where the label indicates whether two instances belong to the same cluster.
• Train a deep neural network on a repository of 100+ datasets from OpenML, using feature vectors derived from clustering metrics and data properties.
• Use a 4-layer feedforward network with ReLU activations and log-softmax output, trained via Adadelta and negative log-likelihood loss over 10 epochs.
• For prediction, average the model’s output for a pair and its reverse to ensure symmetry, and assign to the same cluster if the average probability exceeds 0.5.
• Evaluate performance on internal and external test sets, including data from partitions not used in training to assess generalization.

Experimental results

Research questions

• RQ1 Can unsupervised learning problems be systematically reduced to supervised learning using prior labeled data?
• RQ2 Can a meta-approach improve clustering performance across diverse datasets by learning from prior supervised tasks?
• RQ3 Does leveraging a heterogeneous repository of labeled datasets allow for better selection of clustering algorithms and hyperparameters than heuristic defaults?
• RQ4 Can the framework generalize to unseen data distributions, even when the training data does not include such data?
• RQ5 To what extent can a neural network trained on aggregated, unlabeled data from multiple domains improve unsupervised decision-making?

Key findings

• The meta-approach significantly outperformed a majority rule baseline on both internal and external test sets, with a mean accuracy of 0.75 on meta-ET, demonstrating strong generalization.
• K-means clustering was found to outperform four other algorithms across the repository of datasets, based on data-driven evaluation rather than heuristic preference.
• Systematic biases were identified in a standard heuristic for selecting the number of clusters, and these biases were correctable using meta-learned knowledge.
• The method improved performance in outlier removal by learning optimal thresholds from prior data, enhancing clustering quality.
• The neural network model trained on aggregated, unlabeled data from diverse domains generalized effectively to new, unseen unsupervised problems, even when the test data came from a partition not used in training.
• The framework provides a theoretical foundation that circumvents Kleinberg’s impossibility theorem for clustering by embedding the problem in a meta-distribution over learning tasks.