[Paper Review] Learning over inherently distributed data

Donghui Yan, Ying Xu|arXiv (Cornell University)|Jul 30, 2019

Privacy-Preserving Technologies in Data47 references4 citations

TL;DR

This paper proposes a communication-efficient framework for learning over inherently distributed data by using distortion-minimizing local transformations that require only small local signatures to be shared, enabling parallel local computation and preserving data privacy. The method ensures asymptotically vanishing error as local signature sizes grow, with theoretical guarantees for continuity-based models like linear regression and Random Forests.

ABSTRACT

The recent decades have seen a surge of interests in distributed computing. Existing work focus primarily on either distributed computing platforms, data query tools, or, algorithms to divide big data and conquer at individual machines etc. It is, however, increasingly often that the data of interest are inherently distributed, i.e., data are stored at multiple distributed sites due to diverse collection channels, business operations etc. We propose to enable learning and inference in such a setting via a general framework based on the distortion minimizing local transformations. This framework only requires a small amount of local signatures to be shared among distributed sites, eliminating the need of having to transmitting big data. Computation can be done very efficiently via parallel local computation. The error incurred due to distributed computing vanishes when increasing the size of local signatures. As the shared data need not be in their original form, data privacy may also be preserved. Experiments on linear (logistic) regression and Random Forests have shown promise of this approach. This framework is expected to apply to a general class of tools in learning and inference with the continuity property.

Motivation & Objective

• Address the challenge of performing machine learning on data that are inherently distributed across multiple sites due to organizational, operational, or privacy constraints.
• Overcome the limitations of existing distributed computing methods that assume data can be centrally aggregated or are distributed solely for computational efficiency.
• Develop a framework that enables accurate learning and inference without transmitting raw data, minimizing communication and preserving data privacy.
• Theoretical guarantee that the error from distributed computation vanishes as local signature sizes increase, ensuring statistical consistency.

Proposed method

• Propose a general framework based on distortion-minimizing local transformations to map data from each distributed site into a compact local signature.
• Use space-partitioning trees (e.g., k-d trees) to recursively partition data and ensure shrinking cell diameters, with cuts defined to maintain statistical continuity.
• Define 'good cuts' as those that split node probability mass within a factor of $\sqrt{1+\epsilon}/2$ of the true median, ensuring balanced partitioning.
• Establish that the probability of bad cuts decreases exponentially with node size, leading to full trees up to level $k$ with high probability.
• Leverage the invariance of decision trees under monotonic transformations to assume uniform marginal distributions, enabling diameter shrinkage analysis.
• Prove that the diameter of the cell containing a data point $X$ vanishes in probability as the number of partitioning levels $k$ increases, ensuring convergence.

Experimental results

Research questions

• RQ1 Can we perform accurate learning and inference on inherently distributed data without transmitting raw data across sites?
• RQ2 How can we ensure that the error introduced by distributed computation diminishes as the size of shared local signatures increases?
• RQ3 What theoretical guarantees can be provided for the consistency of learning models under this framework, particularly for models with a continuity property?
• RQ4 How can data privacy be preserved while still enabling effective model training across distributed, non-identical data sources?
• RQ5 What is the role of space-partitioning trees in ensuring convergence of local approximations to global models?

Key findings

• The framework achieves asymptotically vanishing error in learning and inference as the size of local signatures increases, with error $\rightarrow 0$ as $\min(k_1, ..., k_J) \to \infty$.
• The probability of a bad cut in space-partitioning trees decreases exponentially with node size, ensuring that trees become full up to level $k$ with high probability.
• The diameter of the cell containing a data point $X$ shrinks to zero in probability as the number of partitioning levels $k$ increases, due to repeated 'good' cuts.
• The method is communication-efficient, requiring only small local signatures to be shared, and avoids transmission of raw data, thus preserving privacy.
• Experiments demonstrate successful application to linear (logistic) regression and Random Forests, showing practical promise.
• Theoretical analysis confirms that the framework applies to a general class of learning tools with the 'continuity' property, ensuring statistical consistency.