[Paper Review] Evaluation Metrics for Unsupervised Learning Algorithms

Julio-Omar Palacio-Niño, Fernando Berzal|arXiv (Cornell University)|May 14, 2019

Advanced Clustering Algorithms ResearchComputer Science19 references98 citations

TL;DR

A survey of clustering evaluation criteria, detailing Kleinberg’s axioms, internal/external validation metrics, and hyperparameter tuning strategies for unsupervised learning.

ABSTRACT

Determining the quality of the results obtained by clustering techniques is a key issue in unsupervised machine learning. Many authors have discussed the desirable features of good clustering algorithms. However, Jon Kleinberg established an impossibility theorem for clustering. As a consequence, a wealth of studies have proposed techniques to evaluate the quality of clustering results depending on the characteristics of the clustering problem and the algorithmic technique employed to cluster data.

Motivation & Objective

• Motivate the need to assess clustering quality in unsupervised learning where labels are absent.
• Recall Kleinberg’s impossibility theorem and its implications for desirable clustering properties.
• Describe a taxonomy of evaluation criteria (internal and external) for clustering results.
• Survey commonly used internal, external, and information-theoretic validation metrics.
• Outline approaches for hyperparameter tuning in clustering algorithms.

Proposed method

• Discuss Kleinberg’s three axioms: scale invariance, richness, and consistency.
• Present a taxonomy of cluster evaluation methods (internal, external, information-theoretic).
• Describe specific internal validation measures (cohesion, separation, silhouette, CPCC, etc.).
• Describe specific external validation measures (matching sets, peer-to-peer correlation, information-theoretic indices).
• Explain hyperparameter tuning strategies (grid/random/Smart search) and knee/elbow method for nc selection.

Experimental results

Research questions

• RQ1 What are the theoretical limits on clustering quality due to Kleinberg’s axioms?
• RQ2 What evaluation criteria and metrics best capture clustering quality under different algorithmic paradigms?
• RQ3 How can internal, external, and information-theoretic metrics be applied to assess clustering results?
• RQ4 What strategies exist for tuning clustering hyperparameters to improve performance?

Key findings

• Kleinberg’s impossibility theorem shows no clustering function can satisfy all three axioms simultaneously.
• A broad spectrum of internal and external validation metrics exist to assess clustering quality without (internal) or with (external) external information.
• Internal validation focuses on cohesion and separation, with metrics like silhouette and CPCC varying in applicability across algorithm types.
• External validation relies on matching partitions, pairwise correlations, and information-theoretic measures to compare clusterings against reference partitions.
• Hyperparameter tuning for clustering can use grid/random/-smart search strategies, with knee/elbow heuristics aiding the choice of the number of clusters.