[Paper Review] Scalable Support Vector Clustering Using Budget

Tung Pham, Trung Le+1 more|arXiv (Cornell University)|Sep 19, 2017

Anomaly Detection Techniques and Applications8 references3 citations

TL;DR

This paper proposes a scalable support vector clustering method using stochastic gradient descent (SGD) with a budget constraint to overcome the high computational cost of traditional support-based clustering. By modeling the domain of novelty with a hyperplane and employing budgeted SGD, the method achieves comparable clustering quality to baselines while significantly reducing training and clustering times, especially on large datasets.

ABSTRACT

Owing to its application in solving the difficult and diverse clustering or outlier detection problem, support-based clustering has recently drawn plenty of attention. Support-based clustering method always undergoes two phases: finding the domain of novelty and performing clustering assignment. To find the domain of novelty, the training time given by the current solvers is typically over-quadratic in the training size, and hence precluding the usage of support-based clustering method for large-scale datasets. In this paper, we propose applying Stochastic Gradient Descent (SGD) framework to the first phase of support-based clustering for finding the domain of novelty and a new strategy to perform the clustering assignment. However, the direct application of SGD to the first phase of support-based clustering is vulnerable to the curse of kernelization, that is, the model size linearly grows up with the data size accumulated overtime. To address this issue, we invoke the budget approach which allows us to restrict the model size to a small budget. Our new strategy for clustering assignment enables a fast computation by means of reducing the task of clustering assignment on the full training set to the same task on a significantly smaller set. We also provide a rigorous theoretical analysis about the convergence rate for the proposed method. Finally, we validate our proposed method on the well-known datasets for clustering to show that the proposed method offers a comparable clustering quality while simultaneously achieving significant speedup in comparison with the baselines.

Motivation & Objective

• Address the high computational complexity of traditional support-based clustering methods, which scale over-quadratically with dataset size.
• Enable scalable clustering for large-scale datasets by replacing slow optimization solvers with a stochastic gradient descent (SGD) framework.
• Mitigate the curse of kernelization, where model size grows linearly with data, by introducing a budget constraint to limit support vector accumulation.
• Develop a novel clustering assignment strategy that reduces computation by focusing on equilibrium points rather than the full training set.
• Provide a rigorous theoretical convergence analysis for the proposed budgeted SGD-based optimization in the context of support vector clustering.

Proposed method

• Replace the traditional hypersphere-based domain of novelty with a hyperplane to enable optimization via SGD.
• Apply the SGD framework from [33] to solve the optimization problem for finding the optimal hyperplane in the feature space.
• Introduce a budget maintenance procedure (removal, projection, or merging) to restrict the number of support vectors and prevent model size explosion.
• Propose a new clustering assignment strategy where each data point follows a trajectory to an equilibrium point, reducing assignment complexity to a smaller set of equilibria.
• Implement three variants: BSGD-R (removal), BSGD-PNN (projection with k-nearest neighbors), and BSGD-PRN (projection with k-random neighbors) for budget management.
• Theoretical analysis shows convergence of the proposed budgeted SGD method under standard assumptions, ensuring stability and performance.

Experimental results

Research questions

• RQ1 Can SGD-based optimization be effectively applied to the domain-of-novelty phase of support vector clustering to achieve scalability?
• RQ2 How does the budget constraint mitigate the curse of kernelization in kernelized SGD for clustering?
• RQ3 Does the proposed clustering assignment strategy based on equilibrium point trajectories reduce computational cost without sacrificing clustering quality?
• RQ4 How does the proposed method compare to state-of-the-art baselines in terms of clustering accuracy and training speed on real-world datasets?
• RQ5 What is the theoretical convergence rate of the proposed budgeted SGD algorithm in the context of support vector clustering?

Key findings

• The proposed method achieves comparable clustering quality to state-of-the-art baselines across 15 benchmark datasets, with the best purity on 12 out of 15 datasets.
• The BSGD-R variant (removal strategy) is the fastest clustering assignment method, outperforming all baselines in clustering time, especially on large datasets like Musk and Shuttle.
• On the D31 dataset, the proposed method reduced training time from 467.72 seconds (SVC) to 4.58 seconds (BSGD-R), a 100x speedup.
• The method achieved the best DB index on all 15 datasets, indicating superior cluster compactness and separation.
• The BSGD-PNN variant, while slightly slower than BSGD-PRN, achieved the best NMI on 11 of 15 datasets, showing strong performance in normalized mutual information.
• Theoretical analysis confirms the convergence of the proposed budgeted SGD method, providing a solid foundation for its scalability and robustness.