[Paper Review] Learning Entangled Single-Sample Distributions via Iterative Trimming

Hui Yuan, Yingyu Liang|arXiv (Cornell University)|Apr 20, 2020

Machine Learning and Algorithms4 citations

TL;DR

This paper proposes an iterative trimming method for estimating a common parameter in entangled single-sample distributions—where each of $ n $ independent samples comes from a different distribution sharing a common parameter. The method trims noisy points in logarithmic iterations, achieving estimation error dependent only on the noise level of the $ \lceil \alpha n \rceil $-th noisiest point, enabling robustness to a constant fraction (up to 1/5) of high-noise data under general conditions.

ABSTRACT

In the setting of entangled single-sample distributions, the goal is to estimate some common parameter shared by a family of distributions, given one \emph{single} sample from each distribution. We study mean estimation and linear regression under general conditions, and analyze a simple and computationally efficient method based on iteratively trimming samples and re-estimating the parameter on the trimmed sample set. We show that the method in logarithmic iterations outputs an estimation whose error only depends on the noise level of the $\lceil αn ceil$-th noisiest data point where $α$ is a constant and $n$ is the sample size. This means it can tolerate a constant fraction of high-noise points. These are the first such results for the method under our general conditions. It also justifies the wide application and empirical success of iterative trimming in practice. Our theoretical results are complemented by experiments on synthetic data.

Motivation & Objective

• To address parameter estimation in the entangled single-sample distribution setting, where each sample comes from a distinct distribution sharing a common parameter.
• To develop a computationally efficient estimator that remains robust even when a constant fraction of samples are high-noise.
• To theoretically justify the widespread empirical success of iterative trimming in noisy, heterogeneous data settings.
• To extend results beyond symmetric or unimodal distributions to more general conditions, including multivariate mean estimation and linear regression.

Proposed method

• The method iteratively refines an estimate by trimming a fraction of the most deviant samples based on current residuals.
• At each iteration, the algorithm computes a new estimate using only the remaining, less noisy samples.
• The trimming threshold is based on the $ \lceil \alpha n \rceil $-th largest norm of the covariance matrices (for mean estimation) or residual magnitudes (for regression).
• The process runs for $ O(\log n) $ iterations, ensuring computational efficiency.
• For linear regression, the method assumes the design matrix is sufficiently spread out to ensure identifiability.
• The approach is viewed as an efficient, alternating-update variant of the classic trimmed estimator, avoiding exponential search over subsets.

Experimental results

Research questions

• RQ1 Can iterative trimming achieve estimation error that depends only on the noise level of the $ \lceil \alpha n \rceil $-th noisiest point in entangled single-sample distributions?
• RQ2 Does the iterative trimming method maintain robustness when up to a constant fraction of samples are high-noise, even under general distributional assumptions?
• RQ3 Is the method computationally efficient and theoretically justifiable for multivariate mean estimation and linear regression?
• RQ4 How does the performance of iterative trimming compare to oracle estimators that know the true noise levels of all samples?

Key findings

• For multivariate mean estimation, the iterative trimming method achieves an error bound that depends only on the noise level of the $ \lceil \alpha n \rceil $-th largest covariance norm, with $ \alpha \geq 4/5 $, implying robustness to up to 1/5 high-noise points.
• In the univariate and multivariate Gaussian settings, the iterative trimming method achieves estimation error comparable to the oracle mean estimator that uses only the $ \alpha n $ least noisy samples.
• For linear regression, the method achieves error close to the oracle least squares estimator that uses only the $ \alpha n $ samples with smallest noise variances, under a regularity condition on the design matrix.
• The method converges in $ O(\log n) $ iterations, making it computationally efficient compared to exact trimmed estimators.
• Experiments on synthetic data confirm that iterative trimming matches the performance of oracle estimators, validating the theoretical error bounds.
• The results suggest that the radical symmetry assumption in prior work is not necessary, as the method performs well even under asymmetric covariance structures.