[Paper Review] Approximation Algorithms for Clustering via Weighted Impurity Measures

Ferdinando Cicalese, Eduardo Sany Laber|arXiv (Cornell University)|Jul 13, 2018

Gene expression and cancer classification17 references3 citations

TL;DR

This paper presents approximation algorithms for clustering via weighted impurity measures, introducing a linear-time algorithm achieving a 3-approximation for Gini impurity and an $ O(\log \sum_{\mathbf{v} \in V} \|\mathbf{v}\|_1) $-approximation for entropy impurity. It further provides the first polynomial-time $ O(\log^2(\min\{k,L\})) $-approximation algorithm for entropy-based clustering with approximation guarantee independent of the number of data points, and proves the problem is strongly NP-hard for entropy impurity.

ABSTRACT

An impurity measures $I:{R}^k o {R}^+$ maps a $k$-dimensional vector ${\bf v}$ to a non-negative value $I({\bf v})$ so that the more homogeneous ${\bf v}$, the larger its impurity. We study clustering based on impurity measures: given a collection $V$ of $n$ many $k$-dimensional vectors and an impurity measure $I$, the goal is to find a partition ${\cal P}$ of $V$ into $L$ groups $V_1,\ldots,V_L$ that minimizes the total impurities of the groups in ${\cal P}$, i.e., $I({\cal P})= \sum_{m=1}^{L} I(\sum_{{\bf v} \in V_m}{\bf v}).$ Impurity minimization is widely used as quality assessment measure in probability distribution clustering and in categorical clustering where it is not possible to rely on geometric properties of the data set. However, in contrast to the case of metric based clustering, the current knowledge of impurity measure based clustering in terms of approximation and inapproximability results is very limited. Our research contributes to fill this gap. We first present a simple linear time algorithm that simultaneously achieves $3$-approximation for the Gini impurity measure and $O(\log(\sum_{{\bf v} \in V} \| {\bf v} \|_1))$-approximation for the Entropy impurity measure. Then, for the Entropy impurity measure---where we also show that finding the optimal clustering is strongly NP-hard---we are able to design a polynomial time $O(\log^2(\min\{k,L\}))$-approximation algorithm. Our algorithm relies on a nontrivial characterization of a class of clusterings that necessarily includes a partition achieving $O(\log^2(\min\{k,L\}))$--approximation of the impurity of the optimal partition. Remarkably, this is the first polynomial time algorithm with approximation guarantee independent of the number of points/vector and not relying on any restriction on the components of the vectors for producing clusterings with minimum entropy.

Motivation & Objective

• To address the lack of approximation and inapproximability results for clustering based on impurity measures, particularly in categorical and probabilistic data settings.
• To design efficient approximation algorithms for the Partition with Minimum Weighted Impurity Problem (PMWIP), where the goal is to minimize the sum of impurities of clusters formed by aggregating vectors.
• To achieve approximation ratios independent of the number of data points $ n $, especially for entropy impurity, which is critical for large-scale categorical clustering.
• To establish strong NP-hardness for the entropy-based PMWIP, ruling out the possibility of a fully polynomial-time approximation scheme (FPTAS).

Proposed method

• Proposes a simple linear-time algorithm that greedily assigns vectors to clusters to simultaneously achieve a 3-approximation for Gini impurity and an $ O(\log \sum_{\mathbf{v} \in V} \|\mathbf{v}\|_1) $-approximation for entropy impurity.
• Introduces a novel characterization of a class of clusterings that necessarily includes a solution achieving $ O(\log^2(\min\{k,L\})) $-approximation for entropy impurity.
• Employs a bucketing and pruning strategy based on component-wise mass distribution to bound the deviation of partial clusterings from optimal subpartitions.
• Uses the subsystem property of entropy impurity to relate the total impurity to the balance of cluster masses, enabling reduction to the 3-Partition problem.
• Applies a reduction from the strongly NP-hard 3-Partition problem to prove that PMWIP with entropy impurity is strongly NP-hard.

Experimental results

Research questions

• RQ1 Can we design a polynomial-time approximation algorithm for entropy-based clustering with approximation guarantee independent of the number of data points $ n $?
• RQ2 What is the best possible approximation ratio achievable for PMWIP under entropy impurity, and can it be bounded in terms of $ k $ and $ L $?
• RQ3 Is the PMWIP with entropy impurity strongly NP-hard, and does this preclude the existence of an FPTAS?
• RQ4 Can we simultaneously achieve good approximation ratios for both Gini and entropy impurity measures using a single algorithmic framework?

Key findings

• The paper presents a linear-time algorithm that achieves a 3-approximation for Gini impurity and an $ O(\log \sum_{\mathbf{v} \in V} \|\mathbf{v}\|_1) $-approximation for entropy impurity.
• For the case where all vectors have the same $ \ell_1 $-norm, the algorithm achieves an $ O(\log n + \log k) $-approximation for entropy impurity.
• The paper develops a polynomial-time $ O(\log^2(\min\{k,L\})) $-approximation algorithm for entropy impurity, with the approximation factor independent of $ n $.
• The problem of minimizing entropy impurity in PMWIP is proven to be strongly NP-hard via a reduction from the 3-Partition problem.
• The reduction establishes that solving PMWIP with entropy impurity is as hard as solving 3-Partition, implying no FPTAS exists unless P = NP.