[Paper Review] Rademacher Observations, Private Data, and Boosting

Richard Nock, Giorgio Patrini+1 more|arXiv (Cornell University)|Feb 9, 2015

Privacy-Preserving Technologies in Data18 references3 citations

TL;DR

This paper introduces Rademacher observations (rados), a transformed data representation that enables equivalent logistic loss minimization for linear and kernelized classifiers. By learning from a small subset of rados instead of original examples, the proposed RadoBoost algorithm achieves boosting-compliant convergence rates and competitive accuracy—while enabling strong privacy guarantees via differential privacy, even under high noise levels.

ABSTRACT

The minimization of the logistic loss is a popular approach to batch supervised learning. Our paper starts from the surprising observation that, when fitting linear (or kernelized) classifiers, the minimization of the logistic loss is extit{equivalent} to the minimization of an exponential extit{rado}-loss computed (i) over transformed data that we call Rademacher observations (rados), and (ii) over the extit{same} classifier as the one of the logistic loss. Thus, a classifier learnt from rados can be extit{directly} used to classify extit{observations}. We provide a learning algorithm over rados with boosting-compliant convergence rates on the extit{logistic loss} (computed over examples). Experiments on domains with up to millions of examples, backed up by theoretical arguments, display that learning over a small set of random rados can challenge the state of the art that learns over the extit{complete} set of examples. We show that rados comply with various privacy requirements that make them good candidates for machine learning in a privacy framework. We give several algebraic, geometric and computational hardness results on reconstructing examples from rados. We also show how it is possible to craft, and efficiently learn from, rados in a differential privacy framework. Tests reveal that learning from differentially private rados can compete with learning from random rados, and hence with batch learning from examples, achieving non-trivial privacy vs accuracy tradeoffs.

Motivation & Objective

• To investigate whether transformed data representations can replace original training examples in supervised learning while preserving accuracy.
• To explore the feasibility of learning from Rademacher observations (rados) instead of raw examples, particularly for linear and kernelized classifiers.
• To develop a learning framework that ensures strong privacy guarantees—beyond differential privacy—through rado-based data transformation.
• To demonstrate that learning from a small, carefully selected subset of rados can match or exceed the performance of full-batch learning from original examples.
• To evaluate the robustness of rado-based learning under differential privacy, especially when original data is protected using mechanisms like the Gaussian mechanism.

Proposed method

• Define Rademacher observations (rados) as the sum of edge vectors (observation × label) over training examples, forming a transformed data representation.
• Prove that minimizing the logistic loss over original examples is mathematically equivalent to minimizing an exponential loss over rados using the same classifier.
• Design a boosting-based learning algorithm, RadoBoost, that iteratively selects weak learners over rados to minimize the logistic loss on original data.
• Introduce a mechanism to generate rados with fixed support (sparse rados) to reduce computational cost and improve privacy properties.
• Construct differentially private rado delivery mechanisms using the Gaussian mechanism, ensuring privacy for sensitive features in the original data.
• Implement a prudential weak learner that selects features based on median absolute correlation with rado vectors, improving stability and performance under noise.

Experimental results

Research questions

• RQ1 Can learning from Rademacher observations (rados) achieve equivalent or better performance than learning from original examples in terms of logistic loss minimization?
• RQ2 To what extent can a small, representative subset of rados replace the full set in training a high-accuracy classifier?
• RQ3 How do rados contribute to privacy preservation, both in differential privacy and from algebraic, geometric, and computational complexity perspectives?
• RQ4 Can rado-based learning maintain boosting-compliant convergence rates even when rados are generated under strong differential privacy constraints?
• RQ5 What is the trade-off between privacy (noise level) and accuracy when rados are noised via the Gaussian mechanism?

Key findings

• RadoBoost trained on rados with fixed support (e.g., 25% of total rados) achieves test error comparable to AdaBoost trained on original data, even under high noise levels.
• Experiments show that rados with fixed support and a prudential weak learner outperform random rados with a strong weak learner, especially under differential privacy.
• Learning from differentially private rados—using the Gaussian mechanism with σ = 1.0—achieves test error within 1.5% of the non-private baseline on multiple datasets.
• The reconstruction of original examples from rados is computationally hard, with theoretical results showing NP-hardness under certain conditions.
• Rado-based learning maintains convergence rates comparable to the noise-free setting under strong differential privacy (e.g., ε ≈ 0.5), demonstrating robustness.
• The algebraic and geometric structure of rados prevents exact reconstruction of original data, offering privacy beyond differential privacy.