[Paper Review] Wasserstein Measure Coresets

Sebastian Claici, Aude Genevay+1 more|arXiv (Cornell University)|May 18, 2018

Generative Adversarial Networks and Image Synthesis55 references4 citations

TL;DR

This paper introduces Wasserstein measure coresets, a novel framework that generalizes classical discrete coresets by explicitly accounting for the underlying data distribution using optimal transport theory. By minimizing the Wasserstein distance via stochastic gradient descent, the method enables online, memory-efficient coreset construction with strong generalization guarantees across clustering, classification, and Bayesian inference tasks.

ABSTRACT

The proliferation of large data sets and Bayesian inference techniques motivates demand for better data sparsification. Coresets provide a principled way of summarizing a large dataset via a smaller one that is guaranteed to match the performance of the full data set on specific problems. Classical coresets, however, neglect the underlying data distribution, which is often continuous. We address this oversight by introducing Wasserstein measure coresets, an extension of coresets which by definition takes into account generalization. Our formulation of the problem, which essentially consists in minimizing the Wasserstein distance, is solvable via stochastic gradient descent. This yields an algorithm which simply requires sample access to the data distribution and is able to handle large data streams in an online manner. We validate our construction for inference and clustering.

Motivation & Objective

• Address the limitation of classical coresets, which ignore the underlying data distribution and focus only on empirical datasets.
• Formalize a new notion of coresets—measure coresets—that approximate continuous data distributions rather than discrete samples.
• Introduce Wasserstein measure coresets as a principled way to minimize generalization error by leveraging optimal transport theory.
• Develop an online, stochastic optimization algorithm for constructing coresets with minimal memory and computational overhead.
• Demonstrate the versatility of a single coreset across multiple learning tasks, such as clustering, SVM classification, and Bayesian inference.

Proposed method

• Define measure coresets as finitely supported measures that minimize the Wasserstein distance to the true data distribution.
• Formulate coreset construction as a stochastic optimization problem using the Wasserstein distance as the objective function.
• Apply stochastic gradient descent (SGD) to iteratively update coreset points and weights using mini-batches of data, enabling online learning.
• Leverage theoretical connections between optimal transport and coresets to derive generalization error bounds and stability under Lipschitz transformations.
• Use entropy-regularized optimal transport to improve computational efficiency and robustness to noise.
• Adapt the method to different learning tasks by constructing coresets on labeled subsets (e.g., for SVMs) or using them to approximate posterior distributions in Bayesian inference.

Experimental results

Research questions

• RQ1 Can coresets be generalized beyond discrete datasets to account for the underlying continuous data distribution?
• RQ2 How can optimal transport theory be used to define a principled coreset construction that ensures generalization?
• RQ3 Can stochastic gradient descent be effectively used to build coresets in an online, streaming fashion with minimal memory?
• RQ4 How do Wasserstein measure coresets compare to classical coresets and importance sampling methods in terms of performance across diverse learning tasks?
• RQ5 What theoretical guarantees (e.g., generalization error bounds) can be derived for Wasserstein measure coresets under different Wasserstein metrics?

Key findings

• Wasserstein measure coresets significantly outperform uniform sampling in Bayesian inference, reducing KL divergence from the full-data posterior more effectively than baseline methods.
• For $k$-means clustering, $W_2$ coresets achieve performance comparable to the optimal coreset of size 10, demonstrating equivalence to balanced $k$-means optimization.
• The proposed stochastic algorithm enables online coreset construction without storing the full dataset, making it suitable for large-scale and streaming data.
• In SVM classification, coresets constructed via the Wasserstein framework yield better generalization than training on the full dataset for larger coreset sizes, likely due to outlier removal and balanced sampling.
• The method shows robustness to distribution shifts and maintains performance under Lipschitz-continuous loss functions, validating its theoretical stability guarantees.
• While not matching the state-of-the-art in Bayesian inference, the approach still performs substantially better than uniform sampling, indicating strong practical utility.