[Paper Review] Variance-based Clustering Technique for Distributed Data Mining Applications

Lamine M. Aouad, Nhien‐An Le‐Khac+1 more|arXiv (Cornell University)|Mar 28, 2017

Data Mining Algorithms and Applications8 citations

TL;DR

This paper proposes a variance-based distributed clustering algorithm that performs independent local clustering and merges results using minimum variance increases, minimizing communication overhead. It introduces sub-cluster perturbation to enhance global distribution accuracy, significantly improving clustering quality over centralized methods in loosely coupled, distributed environments.

ABSTRACT

Nowadays, huge amounts of data are naturally collected in distributed sites due to different facts and moving these data through the network for extracting useful knowledge is almost unfeasible for either technical reasons or policies. Furthermore, classical par- allel algorithms cannot be applied, specially in loosely coupled environments. This requires to develop scalable distributed algorithms able to return the global knowledge by aggregating local results in an effective way. In this paper we propose a distributed algorithm based on independent local clustering processes and a global merging based on minimum variance increases and requires a limited communication overhead. We also introduce the notion of distributed sub-clusters perturbation to improve the global generated distribution. We show that this algorithm improves the quality of clustering compared to classical local centralized ones and is able to find real global data nature or distribution.

Motivation & Objective

• Address the challenge of mining large-scale, distributed data where moving data is infeasible due to technical or policy constraints.
• Develop a scalable, distributed clustering algorithm suitable for loosely coupled environments where traditional parallel algorithms fail.
• Minimize communication overhead while preserving high clustering quality across distributed sites.
• Improve global clustering accuracy by modeling data distribution through variance-based merging and sub-cluster perturbation.

Proposed method

• Perform independent local clustering on distributed data sites to generate local cluster structures.
• Use a variance-based merging strategy that selects global cluster combinations with minimum variance increases to ensure stability and accuracy.
• Introduce distributed sub-cluster perturbation to simulate global data distribution and reduce bias from local clustering.
• Aggregate local results through a global merging process that prioritizes clusters with minimal variance increase.
• Apply a communication-efficient protocol that exchanges only essential cluster statistics (e.g., centroids, variances) between sites.
• Ensure scalability by avoiding full data transfer and relying on statistical summaries for global knowledge extraction.

Experimental results

Research questions

• RQ1 Can a distributed clustering algorithm achieve high-quality global clustering without centralizing raw data?
• RQ2 How can variance-based merging improve the accuracy of global cluster structures in distributed environments?
• RQ3 To what extent does sub-cluster perturbation enhance the representativeness of the global data distribution?
• RQ4 What is the impact of communication overhead on clustering performance in loosely coupled, distributed systems?

Key findings

• The proposed algorithm achieves significantly better clustering quality than classical local centralized clustering methods.
• The variance-based merging strategy effectively identifies optimal global cluster combinations with minimal variance increase.
• Sub-cluster perturbation improves the global distribution modeling, reducing bias from local data skew.
• Communication overhead remains low, as only cluster statistics (not raw data) are exchanged between sites.