[Paper Review] Tradeoffs for Space, Time, Data and Risk in Unsupervised Learning

Mario Lučić, Mesrob I. Ohannessian+2 more|arXiv (Cornell University)|May 2, 2016

Machine Learning and Algorithms26 references16 citations

TL;DR

This paper introduces a coreset-based framework to navigate tradeoffs among space, time, data size, and statistical risk in unsupervised learning. By strategically summarizing data using coresets, the authors enable faster k-means and Gaussian Mixture Model training with provable risk guarantees, demonstrating that larger datasets reduce running time when combined with intelligent summarization via their TRAM algorithm.

ABSTRACT

Faced with massive data, is it possible to trade off (statistical) risk, and (computational) space and time? This challenge lies at the heart of large-scale machine learning. Using k-means clustering as a prototypical unsupervised learning problem, we show how we can strategically summarize the data (control space) in order to trade off risk and time when data is generated by a probabilistic model. Our summarization is based on coreset constructions from computational geometry. We also develop an algorithm, TRAM, to navigate the space/time/data/risk tradeoff in practice. In particular, we show that for a fixed risk (or data size), as the data size increases (resp. risk increases) the running time of TRAM decreases. Our extensive experiments on real data sets demonstrate the existence and practical utility of such tradeoffs, not only for k-means but also for Gaussian Mixture Models.

Motivation & Objective

• To investigate how data summarization can enable tradeoffs between computational resources (time, space) and statistical risk in unsupervised learning.
• To address the challenge of large-scale learning where increasing data size typically increases computational cost, rather than reducing risk.
• To develop a practical algorithm that navigates the space/time/data/risk tradeoff without relying on oracles.
• To extend computational-statistical tradeoffs beyond convex problems by weakening the data representation instead of the learning algorithm.
• To empirically validate the existence and utility of such tradeoffs in k-means and Gaussian Mixture Models.

Proposed method

• Uses coreset constructions from computational geometry to summarize large datasets into small, representative subsets that preserve solution quality for k-means clustering.
• Applies weighted k-means++ and weighted EM algorithms on coresets to solve the learning problem efficiently on the summarized data.
• Proposes TRAM, a practical algorithm that adaptively adjusts summarization and truncation sizes to balance risk and running time.
• Employs a bisection-like strategy in TRAM: doubling the truncation size and reducing the summarization size by a factor of 1.5 per iteration to explore the tradeoff curve.
• Uses a validation set (1/5 of data) to estimate risk and guide the search for optimal tradeoff points.
• Leverages theoretical coreset bounds to ensure that solutions on coresets approximate the full-data solution within a desired risk tolerance.

Experimental results

Research questions

• RQ1 Can data summarization via coresets enable a tradeoff between running time and risk in unsupervised learning, even as data size increases?
• RQ2 How can we systematically navigate the four-dimensional tradeoff space of space, time, data, and risk in practice?
• RQ3 Does strategic data summarization outperform uniform subsampling in achieving lower risk for the same running time?
• RQ4 Can such tradeoffs be practically realized in non-convex problems like k-means and GMMs?
• RQ5 Is it possible to achieve near-oracle performance in time/risk tradeoffs without exhaustive search?

Key findings

• For a fixed risk tolerance, increasing data size leads to reduced running time when using coresets, demonstrating a data-time tradeoff.
• TRAM achieves running times close to the coreset oracle across all datasets, with only minimal overhead compared to uniform subsampling.
• TRAM outperforms uniform subsampling even though the latter is computationally more expensive to evaluate via exhaustive search.
• For fixed data size, increasing the risk tolerance reduces the required running time, confirming the existence of risk-time tradeoffs.
• The coreset-based approach enables significant speedups—reducing training time from weeks to minutes—while maintaining low risk in GMMs on large datasets like Yahoo! WebScope.
• Empirical results on synthetic, KDD2004BIO, CSN, and WEBSCOPE datasets confirm that coresets dominate uniform subsampling in the time/risk tradeoff space.