[Paper Review] Sample Amplification: Increasing Dataset Size even when Learning is Impossible

Brian Axelrod, Shivam Garg+2 more|arXiv (Cornell University)|Apr 26, 2019

Statistics Education and Methodologies4 citations

TL;DR

This paper introduces sample amplification, a method to generate a larger synthetic dataset from a small original dataset such that the amplified samples are statistically indistinguishable from i.i.d. draws from the true distribution. It shows that for discrete distributions with ≤k elements and d-dimensional Gaussians with fixed covariance, amplification by Θ(n/√k) and Θ(n/√d) samples is possible even when n is too small to learn the distribution, demonstrating that learning is not necessary for faithful data amplification.

ABSTRACT

Given data drawn from an unknown distribution, $D$, to what extent is it possible to ``amplify'' this dataset and output an even larger set of samples that appear to have been drawn from $D$? We formalize this question as follows: an $(n,m)$ $\ ext{amplification procedure}$ takes as input $n$ independent draws from an unknown distribution $D$, and outputs a set of $m > n$ ``samples''. An amplification procedure is valid if no algorithm can distinguish the set of $m$ samples produced by the amplifier from a set of $m$ independent draws from $D$, with probability greater than $2/3$. Perhaps surprisingly, in many settings, a valid amplification procedure exists, even when the size of the input dataset, $n$, is significantly less than what would be necessary to learn $D$ to non-trivial accuracy. Specifically we consider two fundamental settings: the case where $D$ is an arbitrary discrete distribution supported on $\\le k$ elements, and the case where $D$ is a $d$-dimensional Gaussian with unknown mean, and fixed covariance. In the first case, we show that an $\\left(n, n + \\Theta(\\frac{n}{\\sqrt{k}})\ ight)$ amplifier exists. In particular, given $n=O(\\sqrt{k})$ samples from $D$, one can output a set of $m=n+1$ datapoints, whose total variation distance from the distribution of $m$ i.i.d. draws from $D$ is a small constant, despite the fact that one would need quadratically more data, $n=\\Theta(k)$, to learn $D$ up to small constant total variation distance. In the Gaussian case, we show that an $\\left(n,n+\\Theta(\\frac{n}{\\sqrt{d}} )\ ight)$ amplifier exists, even though learning the distribution to small constant total variation distance requires $\\Theta(d)$ samples. In both the discrete and Gaussian settings, we show that these results are tight, to constant factors. Beyond these results, we formalize a number of curious directions for future research along this vein.

Motivation & Objective

• To formalize the problem of generating a larger dataset that is statistically indistinguishable from i.i.d. samples drawn from an unknown distribution D.
• To investigate whether it is possible to amplify a dataset beyond the sample size required to learn D to non-trivial accuracy.
• To establish tight bounds on the maximum amplification possible in two fundamental settings: bounded discrete distributions and high-dimensional Gaussians with fixed covariance.
• To demonstrate that amplification is possible even when the input sample size n is significantly smaller than the sample complexity required for distribution learning.
• To explore theoretical and practical implications of data amplification, including applications in generative modeling and robustness testing.

Proposed method

• Formalize sample amplification as an (n,m) procedure where a function f maps n samples to m > n samples, such that the output's total variation distance from m i.i.d. draws from D is ≤1/3.
• Use a verifier-based framework: a distinguisher that checks for statistical anomalies (e.g., number of unique elements) to test if the amplified set is real or synthetic.
• Analyze the expected number of unique elements E[U^n] and its concentration via concentration inequalities (e.g., Hoeffding-type bounds) to show that amplifiers producing too many or too few unique items can be detected.
• Construct an amplifier that leverages the empirical distribution of the input n samples and adds new samples by sampling from a smoothed version of the empirical distribution, ensuring statistical indistinguishability.
• Prove tightness of amplification bounds by constructing a verifier that detects deviations in the number of unique samples, showing that any amplifier exceeding the bound can be distinguished with high probability.
• Apply Chebyshev and Chernoff-type bounds to establish concentration of the number of unique samples in both the input and amplified sets, enabling the verifier to distinguish real from synthetic data with high confidence.

Experimental results

Research questions

• RQ1 Can we generate a larger dataset from a small number of samples such that the amplified data is statistically indistinguishable from i.i.d. samples from the true distribution D?
• RQ2 Is it possible to amplify a dataset even when the number of input samples is too small to learn D to non-trivial accuracy?
• RQ3 What is the maximum number of samples we can amplify by, in terms of n and the complexity of D (e.g., k for discrete, d for Gaussian)?
• RQ4 Can we prove that the proposed amplification bounds are tight, to within constant factors, in both the discrete and Gaussian settings?
• RQ5 What are the implications of such amplification for generative modeling, distribution testing, and robustness evaluation in machine learning?

Key findings

• An (n, n + Θ(n/√k)) amplification procedure exists for any discrete distribution with at most k support elements, even when n = O(√k), which is quadratically smaller than the Θ(k) samples needed to learn D.
• For d-dimensional Gaussians with fixed covariance, an (n, n + Θ(n/√d)) amplification procedure exists, despite the fact that Θ(d) samples are required to learn the distribution.
• The amplification bounds are tight to within constant factors: no (n, n + ω(n/√k)) amplifier exists for discrete distributions, and no (n, n + ω(n/√d)) amplifier exists for Gaussians.
• The amplification is achieved without learning D; the amplifier only uses the empirical distribution of the input samples and adds new samples via a carefully designed smoothing mechanism.
• A verifier can distinguish amplified data from real i.i.d. samples with high probability if the number of unique elements deviates significantly from expectation, proving the tightness of the bounds.
• The results show that statistical indistinguishability can be achieved with far fewer samples than required for distribution learning, revealing a fundamental separation between learning and amplification.