[Paper Review] Asymptotics of Lower Dimensional Zero-Density Regions

Hengrui Luo, Steve MacEachern+1 more|arXiv (Cornell University)|Jun 3, 2020

Topological and Geometric Data Analysis34 references4 citations

TL;DR

This paper proposes a method to asymptotically detect lower-dimensional zero-density regions in data using covering balls with shrinking radii, leveraging distributional properties of the underlying density. It proves that under specific conditions on the radius decay and density smoothness, such regions can be identified with probability approaching one as sample size increases, while minimizing false positives.

ABSTRACT

Topological data analysis (TDA) allows us to explore the topological features of a dataset. Among topological features, lower dimensional ones have recently drawn the attention of practitioners in mathematics and statistics due to their potential to aid the discovery of low dimensional structure in a data set. However, lower dimensional features are usually challenging to detect based on finite samples and using TDA methods that ignore the probabilistic mechanism that generates the data. In this paper, lower dimensional topological features occurring as zero-density regions of density functions are introduced and thoroughly investigated. Specifically, we consider sequences of coverings for the support of a density function in which the coverings are comprised of balls with shrinking radii. We show that, when these coverings satisfy certain sufficient conditions as the sample size goes to infinity, we can detect lower dimensional, zero-density regions with increasingly higher probability while guarding against false detection. We supplement the theoretical developments with the discussion of simulated experiments that elucidate the behavior of the methodology for different choices of the tuning parameters that govern the construction of the covering sequences and characterize the asymptotic results.

Motivation & Objective

• To address the challenge of detecting lower-dimensional, zero-density regions in data that are invisible to standard topological data analysis (TDA) due to their measure-zero support.
• To develop a method that uses the asymptotic behavior of data accumulation near such regions, rather than finite-sample point patterns.
• To ensure high detection probability for zero-density regions while guarding against false detection of non-existent features.
• To formalize conditions under which covering ball constructions can reliably identify these features as sample size increases.

Proposed method

• Constructs a sequence of coverings of the support of a density function using balls with radii shrinking as sample size increases.
• Focuses on balls intersecting a target zero-density region $ S_0 $, aiming for all such balls to be empty of data points in the limit.
• Uses a probabilistic bound based on the expected number of points in $ ho(n) $-neighborhoods of $ S_0 $, derived from the density's local behavior.
• Applies the Bernoulli inequality to show that the probability of all relevant balls being empty tends to one under sufficient decay conditions on the radius.
• Defines a critical condition involving the dimension $ d $, the lower-dimensional structure $ d_0 $, and the Hölder smoothness $ \underline{K}_f $ of the density: $ 1 + d_0\eta - \underline{K}_f\eta - d\eta < 0 $.
• Introduces a tuning parameter $ \epsilon(n) $ to define neighborhood size around $ S_0 $, and ensures the radius $ r(n) $ decays at a rate compatible with this neighborhood and the smoothness of $ f $.

Experimental results

Research questions

• RQ1 Can zero-density regions of lower dimension be detected with increasing accuracy as sample size grows, despite having no probability mass?
• RQ2 What conditions on the radius decay and density smoothness ensure that covering balls around such regions remain empty with high probability?
• RQ3 How can the method distinguish true zero-density structures from spurious holes or noise in finite samples?
• RQ4 What role does the local Hölder smoothness of the density play in the detectability of lower-dimensional features?
• RQ5 How do the choice of covering radius and neighborhood size affect the asymptotic detection probability?

Key findings

• Under the condition $ 1 + d_0\eta - \underline{K}_f\eta - d\eta < 0 $, the probability that all $ \epsilon(n) $-inside covering balls intersecting $ S_0 $ are empty converges to one as $ n \to \infty $.
• The method ensures that non-zero-density regions are eventually covered by balls containing at least one data point, preserving feature distinction.
• The asymptotic detection rate depends on the interplay between the dimension $ d $, the lower-dimensional structure $ d_0 $, and the local Hölder smoothness $ \underline{K}_f $ of the density.
• The probability of false detection is controlled by ensuring the expected number of points in relevant balls decays sufficiently fast.
• The theoretical framework is supported by simulated experiments showing behavior across different tuning parameters for radius and neighborhood size.
• The result holds for well-behaved densities on $[0,1]^d$, and extends to more general supports under mild regularity conditions.