[Paper Review] Robust Coreset Construction for Distributed Machine Learning

Hanlin Lu, Ming-Ju Li+4 more|arXiv (Cornell University)|Apr 11, 2019

Face and Expression Recognition41 references4 citations

TL;DR

This paper proposes robust coreset construction algorithms for distributed machine learning that generate a single, reusable coreset capable of supporting diverse supervised and unsupervised learning problems—such as k-means, PCA, SVM, and neural networks—while minimizing communication overhead. By leveraging weighted k-means/median clustering centers and distributing coreset construction across nodes, the method achieves provable approximation guarantees across multiple models with minimal data transfer, validated through extensive experiments on real-world datasets including MNIST and HAR.

ABSTRACT

Coreset, which is a summary of the original dataset in the form of a small weighted set in the same sample space, provides a promising approach to enable machine learning over distributed data. Although viewed as a proxy of the original dataset, each coreset is only designed to approximate the cost function of a specific machine learning problem, and thus different coresets are often required to solve different machine learning problems, increasing the communication overhead. We resolve this dilemma by developing robust coreset construction algorithms that can support a variety of machine learning problems. Motivated by empirical evidence that suitably-weighted k-clustering centers provide a robust coreset, we harden the observation by establishing theoretical conditions under which the coreset provides a guaranteed approximation for a broad range of machine learning problems, and developing both centralized and distributed algorithms to generate coresets satisfying the conditions. The robustness of the proposed algorithms is verified through extensive experiments on diverse datasets with respect to both supervised and unsupervised learning problems.

Motivation & Objective

• Address the high communication cost in distributed machine learning caused by constructing problem-specific coresets for each learning task.
• Develop a unified coreset construction framework that supports a broad range of machine learning problems—including both supervised and unsupervised models—using a single coreset.
• Ensure theoretical approximation guarantees for the coreset across diverse learning problems, including k-means, PCA, SVM, and deep neural networks.
• Design efficient centralized and distributed algorithms that scale to large, high-dimensional datasets while preserving data privacy and minimizing bandwidth usage.
• Verify the robustness of the coreset across varying data distributions, model complexities, and communication constraints in real-world edge computing scenarios.

Proposed method

• Construct coresets using suitably weighted k-means or k-median clustering centers as the core summarization primitive, leveraging empirical evidence of their robustness.
• Establish theoretical conditions under which these weighted centers form an ϵ-coreset that guarantees bounded relative approximation error for a wide class of learning problems.
• Design a distributed coreset construction algorithm (DRCC) that allocates coreset size and local centers across nodes to balance accuracy and communication cost.
• Introduce a communication-efficient coreset construction (CDCC) protocol that enables edge nodes to collaboratively build a global coreset with minimal data transfer.
• Apply Lipschitz continuity bounds to extend approximation guarantees from basic problems (e.g., MEB) to more complex models like SVM and neural networks.
• Use Monte Carlo evaluation to empirically validate the relative approximation error bounds and ensure they remain within theoretical limits across datasets and models.

Figure 1: Application scenario (ML $i$ : machine learning model $i$ ).

Experimental results

Research questions

• RQ1 Can a single coreset be constructed to support multiple machine learning problems in a distributed setting without re-collecting data?
• RQ2 What theoretical conditions ensure that a coreset built from k-means/median centers provides a provable approximation for diverse learning problems?
• RQ3 How can coreset size and local center allocation be optimized across distributed nodes to maintain robustness across different data distributions and model types?
• RQ4 To what extent does the proposed coreset maintain accuracy for complex models like deep neural networks and SVMs compared to training on the full dataset?
• RQ5 How do coreset size and number of local centers affect model performance in distributed coreset construction, and what trade-offs exist?

Key findings

• The proposed coreset construction achieves a maximum relative approximation error of 0.0053 on Fisher’s Iris dataset (MEB, coreset size 20), well below the theoretical bound ε = 0.1073.
• For MNIST, the coreset achieved a relative error of 0.0024 with a theoretical bound ε = 10.74, demonstrating that actual error is significantly lower than the worst-case bound.
• Increasing the number of local centers (K) from 1 to 2 improved model performance notably, especially for neural networks, but further increases yielded diminishing returns for unsupervised models.
• The distributed coreset construction (DRCC) outperformed baseline methods in model quality when coreset size was increased, showing strong scalability and communication efficiency.
• The coreset enabled training of deep neural networks on HAR and MNIST with accuracy within 78.01% and 87.01% of the full-data model, respectively, despite transferring only a tiny fraction of raw data.
• Theoretical bounds on approximation error were consistently upper-bounded across all datasets and models, confirming the robustness of the coreset construction across diverse learning problems.

Figure 2: Comparison of coreset construction algorithms (coreset size: 8).