[Paper Review] Sample-Optimal Density Estimation in Nearly-Linear Time

Clément L. Canonne, Ilias Diakonikolas+2 more|arXiv (Cornell University)|Jun 1, 2015

Complexity and Algorithms in Graphs3 references26 citations

TL;DR

This paper presents a new algorithm for agnostic density estimation of univariate distributions that are well-approximated by piecewise polynomials. It achieves sample-optimal performance with O(t(d+1)/ϵ²) samples and runs in nearly-linear time eO(n·poly(d)), outputting a hypothesis that is 4·OPT + ϵ close to the true density with high probability. The method combines iterative partitioning, efficient polynomial fitting via separation oracles, and a combinatorial algorithm for hyperplane separation, enabling the first nearly sample-optimal and nearly-linear time estimators for a broad class of structured distributions including mixtures of Gaussians, log-concave, and discrete distributions.

ABSTRACT

We design a new, fast algorithm for agnostically learning univariate probability distributions whose densities are well approximated by piecewise polynomial functions. Let $f$ be the density function of an arbitrary univariate distribution, and suppose that $f$ is $\mathrm{OPT}$-close in $L_1$-distance to an unknown piecewise polynomial function with $t$ interval pieces and degree $d$. Our algorithm draws $n = O(t(d+1)/\epsilon^2)$ samples from $f$, runs in time $ ilde{O}(n \cdot \mathrm{poly}(d))$, and with probability at least $9/10$ outputs an $O(t)$-piecewise degree-$d$ hypothesis $h$ that is $4 \cdot \mathrm{OPT} +\epsilon$ close to $f$. Our general algorithm yields (nearly) sample-optimal and nearly-linear time estimators for a wide range of structured distribution families over both continuous and discrete domains in a unified way. For most of our applications, these are the first sample-optimal and nearly-linear time estimators in the literature. As a consequence, our work resolves the sample and computational complexities of a broad class of inference tasks via a single "meta-algorithm". Moreover, we experimentally demonstrate that our algorithm performs very well in practice. Our algorithm consists of three "levels": (i) At the top level, we employ an iterative greedy algorithm for finding a good partition of the real line into the pieces of a piecewise polynomial. (ii) For each piece, we show that the sub-problem of finding a good polynomial fit on the current interval can be solved efficiently with a separation oracle method. (iii) We reduce the task of finding a separating hyperplane to a combinatorial problem and give an efficient algorithm for this problem. Combining these three procedures gives a density estimation algorithm with the claimed guarantees.

Motivation & Objective

• To design a fast, agnostic density estimation algorithm that is both statistically and computationally efficient for univariate distributions.
• To achieve sample-optimal performance for structured distribution families such as mixtures of Gaussians, log-concave, and discrete distributions.
• To resolve the long-standing open problem of simultaneously minimizing sample complexity and running time for density estimation in the agnostic setting.
• To unify the design of estimators across continuous and discrete domains using a single meta-algorithm.

Proposed method

• The algorithm uses an iterative greedy merging procedure to partition the real line into intervals suitable for piecewise polynomial fitting.
• For each interval, it employs a separation oracle method to efficiently compute the best-fitting polynomial under L1 constraints.
• It reduces the problem of finding a separating hyperplane to a combinatorial optimization problem, which is solved with a custom algorithm.
• The method leverages an Ak-projection oracle to compute polynomial projections efficiently, enabling fast convergence.
• The overall framework is modular and applies uniformly to both continuous and discrete domains.
• It uses a novel reduction from polynomial fitting to a spectrahedral feasibility problem, solvable via convex optimization.

Experimental results

Research questions

• RQ1 Can we design a density estimation algorithm that is both sample-optimal and runs in nearly-linear time for univariate distributions?
• RQ2 Is it possible to achieve agnostic learning guarantees for structured families like mixtures of Gaussians and log-concave distributions using a unified meta-algorithm?
• RQ3 What is the computational cost of learning piecewise polynomial approximations, and can it be reduced to nearly-linear time?
• RQ4 How does the algorithm’s performance scale with sample size and complexity of the target distribution?
• RQ5 Can the separation oracle approach be adapted to handle non-negativity and boundedness constraints efficiently in polynomial fitting?

Key findings

• The algorithm achieves sample complexity of O(t(d+1)/ϵ²), which is information-theoretically optimal for distributions well-approximated by t-piecewise degree-d polynomials.
• It runs in time eO(n·poly(d)), where n is the number of samples, making it nearly-linear in sample size.
• For 10⁶ samples, the algorithm runs in under 35 milliseconds for histogram hypotheses and under 0.3 seconds for piecewise linear hypotheses, outperforming sorting-based methods.
• The learning error decays as O(t(d+1)/ϵ²), and the constant factor in the error bound is close to 1, indicating near-optimality.
• The running time is robust and nearly independent of the underlying distribution, demonstrating strong practical performance.
• The method yields the first sample-optimal and nearly-linear time estimators for mixtures of Gaussians, log-concave, t-monotone, and Besov-space densities.