[Paper Review] Differentially Private k-Means Clustering with Guaranteed Convergence

Zhigang Lü, Hong Shen|arXiv (Cornell University)|Feb 3, 2020

Privacy-Preserving Technologies in Data28 references4 citations

TL;DR

This paper proposes a novel differentially private k-means clustering framework that ensures convergence by injecting noise within a dynamically defined convergence zone around perturbed centroids, using both past and future centroid movement directions. It proves convergence in at most twice the iterations of Lloyd’s algorithm and achieves superior clustering quality under the same privacy budget compared to state-of-the-art methods.

ABSTRACT

Iterative clustering algorithms help us to learn the insights behind the data. Unfortunately, this may allow adversaries to infer the privacy of individuals with some background knowledge. In the worst case, the adversaries know the centroids of an arbitrary iteration and the information of n-1 out of n items. To protect individual privacy against such an inference attack, preserving differential privacy (DP) for the iterative clustering algorithms has been extensively studied in the interactive settings. However, existing interactive differentially private clustering algorithms suffer from a non-convergence problem, i.e., these algorithms may not terminate without a predefined number of iterations. This problem severely impacts the clustering quality and the efficiency of a differentially private algorithm. To resolve this problem, in this paper, we propose a novel differentially private clustering framework in the interactive settings which controls the orientation of the movement of the centroids over the iterations to ensure the convergence by injecting DP noise in a selected area. We prove that, in the expected case, algorithm under our framework converges in at most twice the iterations of Lloyd's algorithm. We perform experimental evaluations on real-world datasets to show that our algorithm outperforms the state-of-the-art of the interactive differentially private clustering algorithms with guaranteed convergence and better clustering quality to meet the same DP requirement.

Motivation & Objective

• To address the non-convergence problem in existing differentially private k-means algorithms that fail to terminate within a bounded number of iterations.
• To ensure convergence of differentially private k-means clustering in the interactive setting while preserving strong privacy guarantees.
• To improve clustering quality under the same differential privacy budget by leveraging temporal knowledge of centroid movement.
• To provide theoretical convergence bounds and empirical validation of the proposed method on real-world datasets.

Proposed method

• The method maintains perturbed centroids from the previous iteration to define a convergence zone in the current iteration, within which differential privacy noise is injected.
• It uses two strategies to control the orientation of centroid movement: one based on past movement (from t−1 to t) and another incorporating future movement (from t to t+1) for improved convergence.
• Noise injection is performed using the exponential mechanism within the convergence zone to ensure differential privacy while maintaining convergence.
• The algorithm dynamically adjusts the noise injection region based on the centroid movement trajectory, ensuring stability and convergence.
• Theoretical analysis proves that the algorithm converges in at most twice the number of iterations as Lloyd’s algorithm in expectation.
• The method integrates both past and future knowledge to increase the probability of converging to the same solution as Lloyd’s algorithm under the same initial centroids.

Experimental results

Research questions

• RQ1 Can differentially private k-means clustering be made to converge reliably in the interactive setting, unlike existing methods that suffer from non-convergence?
• RQ2 How can noise injection be strategically guided to preserve convergence while ensuring differential privacy?
• RQ3 What is the impact of using past versus future centroid movement information on convergence speed and solution fidelity?
• RQ4 Can the proposed method achieve better clustering quality than state-of-the-art differentially private k-means algorithms under the same privacy budget?
• RQ5 Does the use of both past and future knowledge significantly improve convergence probability compared to using only past knowledge?

Key findings

• The proposed algorithm converges to a solution of Lloyd’s algorithm in at most twice the number of iterations as Lloyd’s algorithm in expectation.
• When using both past and future knowledge, the algorithm converges to the same solution as Lloyd’s algorithm with high probability, matching at least 84% of the output centroids in practice.
• The posterior strategy (using only past knowledge) matches no more than 80% of Lloyd’s centroids, while the prior strategy (using both past and future knowledge) achieves at least 84% match rate.
• In terms of iteration count, the prior strategy requires 1.22 to 1.40 times more iterations than Lloyd’s algorithm, while the posterior strategy requires only 0.95 to 0.97 times, indicating a trade-off between convergence quality and speed.
• On six real-world datasets, the proposed method outperforms state-of-the-art differentially private clustering algorithms in clustering quality under the same privacy budget.
• The experimental results confirm the theoretical analysis, showing that the iteration ratio remains stable across different privacy budgets, with the prior strategy consistently requiring more iterations due to additional future knowledge computation.