[Paper Review] Density Estimation via Discrepancy Based Adaptive Sequential Partition

Dangna Li, Kun Yang+1 more|PubMed|Apr 5, 2014

Machine Learning and Algorithms21 references10 citations

TL;DR

This paper proposes a nonparametric density estimation method using discrepancy-based adaptive sequential partitioning to construct a piecewise constant density function on a binary partition of the domain. By leveraging star discrepancy from Quasi-Monte Carlo theory, the method achieves an optimal $O(n^{-1/2})$ convergence rate, outperforms kernel density estimation in high dimensions, and enables efficient applications such as k-means initialization and level set tree construction.

ABSTRACT

Given <i>iid</i> observations from an unknown absolute continuous distribution defined on some domain Ω, we propose a nonparametric method to learn a piecewise constant function to approximate the underlying probability density function. Our density estimate is a piecewise constant function defined on a binary partition of Ω. The key ingredient of the algorithm is to use discrepancy, a concept originates from Quasi Monte Carlo analysis, to control the partition process. The resulting algorithm is simple, efficient, and has a provable convergence rate. We empirically demonstrate its efficiency as a density estimation method. We also show how it can be utilized to find good initializations for k-means.

Motivation & Objective

• To address the limitations of parametric and kernel density estimation in high-dimensional settings, where bandwidth sensitivity and computational infeasibility arise.
• To develop a nonparametric, efficient, and provably convergent method for density estimation using adaptive domain partitioning.
• To integrate concepts from Quasi-Monte Carlo analysis—specifically star discrepancy—into density estimation for improved uniformity control and convergence.
• To provide a scalable alternative to Bayesian nonparametric and tree-based methods that are computationally expensive or rely on complex posterior sampling.
• To demonstrate practical utility in real-world tasks such as k-means initialization, image segmentation, and hierarchical data visualization.

Proposed method

• The method constructs a piecewise constant density estimate on a binary partition of the domain $\Omega = [0,1]^d$, where each sub-rectangle is assigned a constant density value.
• It uses star discrepancy as a formal measure of uniformity within each sub-rectangle to guide the recursive partitioning process, ensuring balanced and data-adaptive splits.
• The algorithm proceeds top-down in a greedy fashion, splitting sub-rectangles only when the discrepancy exceeds a threshold, thus controlling approximation error.
• The partitioning is adaptive: splits are not restricted to midpoints, but are determined by a 'gap' statistic that reflects data distribution, improving representational efficiency.
• The final density estimate is a normalized piecewise constant function over the resulting partition, with each region's density computed as the empirical frequency of observations.
• The method leverages the fact that for any piecewise constant density, the conditional distribution within each region is uniform, enabling discrepancy to serve as a natural error metric.

Experimental results

Research questions

• RQ1 Can discrepancy from Quasi-Monte Carlo methods be effectively used to guide adaptive, nonparametric density estimation in high-dimensional spaces?
• RQ2 Does a discrepancy-based partitioning strategy yield a density estimator with provably optimal convergence rate?
• RQ3 How does the proposed method compare in accuracy and speed to kernel density estimation and other nonparametric methods in high-dimensional settings?
• RQ4 Can the resulting piecewise constant density function be used effectively to initialize k-means clustering?
• RQ5 Can the method support hierarchical data analysis tasks such as level set tree construction and multi-level feature extraction?

Key findings

• The proposed method achieves an $O(n^{-1/2})$ convergence rate in $L^2$ error, which is optimal for Monte Carlo-type methods and matches the theoretical lower bound for nonparametric density estimation.
• Empirically, the method achieves comparable Hellinger distance to kernel density estimation while being approximately 100 times faster, demonstrating significant computational efficiency.
• The method successfully finds good initializations for k-means clustering, improving convergence speed and solution quality in experiments.
• The piecewise constant density estimate enables effective image segmentation by grouping pixels based on joint color and spatial features, outperforming mean shift in visual quality and stability.
• The method supports the construction of level set trees (connectivity graphs) from data alone, revealing hierarchical structure and modes in high-dimensional data with minimal noise.
• Multi-level feature extraction using densities from different partition depths captures non-linear data structure, showing promise for representation learning and downstream tasks.