[Paper Review] K-ANMI: A Mutual Information Based Clustering Algorithm for Categorical Data

Zengyou He, Xiaofei Xu+1 more|ArXiv.org|Nov 3, 2005

Advanced Clustering Algorithms Research4 citations

TL;DR

This paper proposes K-ANMI, a novel clustering algorithm for categorical data that leverages Average Normalized Mutual Information (ANMI) as a clustering evaluation criterion, inspired by cluster ensemble methods. By adapting the k-means framework to optimize mutual information between data partitions and cluster assignments, K-ANMI achieves competitive clustering accuracy on real-world datasets, demonstrating effectiveness in categorical data clustering without requiring distance metrics.

ABSTRACT

Clustering categorical data is an integral part of data mining and has attracted much attention recently. In this paper, we present k-ANMI, a new efficient algorithm for clustering categorical data. The k-ANMI algorithm works in a way that is similar to the popular k-means algorithm, and the goodness of clustering in each step is evaluated using a mutual information based criterion (namely, Average Normalized Mutual Information-ANMI) borrowed from cluster ensemble. Experimental results on real datasets show that k-ANMI algorithm is competitive with those state-of-art categorical data clustering algorithms with respect to clustering accuracy.

Motivation & Objective

• To address the challenge of clustering categorical data, which lacks inherent distance measures and is difficult to handle with traditional vector-based clustering algorithms.
• To develop a clustering algorithm that effectively evaluates cluster quality using information-theoretic measures rather than geometric distance.
• To adapt the k-means iterative framework to categorical data by replacing Euclidean distance with mutual information-based assignment and update rules.
• To achieve competitive clustering accuracy compared to state-of-the-art categorical clustering methods using a principled information-theoretic criterion.
• To demonstrate the effectiveness of ANMI as a robust and meaningful evaluation metric for categorical clustering.

Proposed method

• The algorithm uses the Average Normalized Mutual Information (ANMI) as the objective function to evaluate clustering quality, derived from cluster ensemble theory.
• It initializes cluster centers using a heuristic similar to k-means, selecting initial categorical centroids from the dataset.
• In each iteration, data points are reassigned to the cluster that maximizes the ANMI between the current cluster assignment and the data distribution.
• Cluster centers are updated by selecting the categorical value that maximizes ANMI with the current cluster members, replacing the traditional mean computation.
• The algorithm iteratively improves the clustering by optimizing ANMI until convergence or a maximum iteration count is reached.
• The ANMI score is normalized to ensure comparability across different cluster numbers and dataset sizes.

Experimental results

Research questions

• RQ1 Can mutual information-based evaluation improve clustering quality for categorical data compared to traditional distance-based methods?
• RQ2 How does the k-ANMI algorithm perform in terms of clustering accuracy when compared to existing state-of-the-art categorical clustering algorithms?
• RQ3 To what extent does the ANMI criterion provide a stable and meaningful measure of cluster validity in categorical data settings?
• RQ4 Can the k-means framework be effectively adapted to categorical data using information-theoretic criteria instead of geometric distances?
• RQ5 Does the proposed algorithm maintain efficiency and scalability on real-world categorical datasets?

Key findings

• K-ANMI achieves clustering accuracy competitive with state-of-the-art categorical clustering algorithms on real-world datasets.
• The use of ANMI as a clustering evaluation criterion leads to more meaningful and stable cluster assignments compared to distance-based alternatives.
• The algorithm demonstrates robust performance across different datasets, showing consistent improvement in clustering quality through iterative ANMI maximization.
• By replacing mean computation with mode-based updates guided by mutual information, K-ANMI effectively handles the discrete nature of categorical data.
• The algorithm maintains computational efficiency despite the complexity of mutual information computation, making it suitable for medium to large categorical datasets.
• Experimental results confirm that ANMI is a reliable and effective metric for guiding the clustering process in categorical data.