[Paper Review] Efficient Dynamic Clustering: Capturing Patterns from Historical Cluster Evolution

Binbin Gu, Saeed Kargar+1 more|arXiv (Cornell University)|Jan 1, 2022

Data Stream Mining Techniques6 citations

TL;DR

This paper proposes DynamicC, a machine learning-augmented system that learns cluster evolution patterns from historical batch clustering results to enable fast, accurate dynamic clustering in high-velocity data environments. By training on how clusters change in response to data updates, DynamicC predicts merge/split decisions and applies heuristics using the objective function, achieving near-batch accuracy with significantly lower latency compared to state-of-the-art methods.

ABSTRACT

Clustering aims to group unlabeled objects based on similarity inherent among them into clusters. It is important for many tasks such as anomaly detection, database sharding, record linkage, and others. Some clustering methods are taken as batch algorithms that incur a high overhead as they cluster all the objects in the database from scratch or assume an incremental workload. In practice, database objects are updated, added, and removed from databases continuously which makes previous results stale. Running batch algorithms is infeasible in such scenarios as it would incur a significant overhead if performed continuously. This is particularly the case for high-velocity scenarios such as ones in Internet of Things applications. In this paper, we tackle the problem of clustering in high-velocity dynamic scenarios, where the objects are continuously updated, inserted, and deleted. Specifically, we propose a generally dynamic approach to clustering that utilizes previous clustering results. Our system, DynamicC, uses a machine learning model that is augmented with an existing batch algorithm. The DynamicC model trains by observing the clustering decisions made by the batch algorithm. After training, the DynamicC model is usedin cooperation with the batch algorithm to achieve both accurate and fast clustering decisions. The experimental results on four real-world and one synthetic datasets show that our approach has a better performance compared to the state-of-the-art method while achieving similarly accurate clustering results to the baseline batch algorithm.

Motivation & Objective

• To address the high computational overhead of re-running batch clustering algorithms after every data update in dynamic databases.
• To design a general-purpose dynamic clustering system that works with any batch clustering algorithm without relying on algorithm-specific assumptions.
• To reduce latency in high-velocity workloads such as IoT and cloud applications by learning predictable patterns in cluster evolution.
• To maintain clustering quality comparable to batch methods while drastically improving performance through learned prediction.
• To enable continuous retraining and model adaptation for evolving data workloads.

Proposed method

• DynamicC trains a machine learning model on historical cluster evolution data generated by a batch clustering algorithm in response to insertions, deletions, and modifications.
• The system represents clustering changes using compact binary decisions (merge or split) and global similarity metrics to reduce dimensionality and enable efficient learning.
• During prediction, the model outputs general merge/split decisions, which are then mapped to specific node movements using heuristics based on the clustering objective function.
• The system validates predictions by checking if the change improves the objective function score, rejecting inaccurate predictions to maintain quality.
• DynamicC supports continuous retraining to adapt to evolving data patterns and maintain model accuracy over time.
• The method is agnostic to the underlying batch algorithm, enabling integration with k-means, DBSCAN, and other clustering techniques with minimal changes.

Experimental results

Research questions

• RQ1 Can a machine learning model learn and predict cluster evolution patterns from historical batch clustering results in dynamic databases?
• RQ2 How can cluster evolution be represented compactly and effectively to enable efficient training and prediction?
• RQ3 Can a learned model achieve clustering accuracy comparable to batch methods while reducing latency in high-velocity workloads?
• RQ4 How does the system ensure prediction correctness when the model's output may be inaccurate?
• RQ5 To what extent can the model be retrained continuously to adapt to evolving data patterns?

Key findings

• DynamicC achieves F1 scores of 0.997 on the Cora dataset, 0.994 on Music, and 0.992 on the Synthetic dataset—comparable to the baseline batch algorithm.
• On the Cora dataset, DynamicC reduces latency significantly compared to state-of-the-art methods while maintaining near-perfect F1 scores.
• The Logistic Regression model reaches 98% recall and 98% accuracy when trained on 80% of the Cora dataset, demonstrating strong learning convergence.
• Training time for all ML models (Logistic Regression, SVM, Decision Tree) remains under 1 second even with 20,000 training samples.
• The system’s heuristic-based action mapping ensures that only predictions improving the objective function are applied, preserving clustering quality.
• DynamicC enables continuous retraining, allowing the model to adapt to long-term data drift and maintain performance over time.