[Paper Review] Nonparametric method for space conditional density estimation in moderately large dimensions

Minh-Lien Jeanne Nguyen|arXiv (Cornell University)|Jan 19, 2018

Statistical Methods and Inference16 references3 citations

TL;DR

This paper proposes a greedy, nonparametric kernel-based method for conditional density estimation in moderately high dimensions, using a revised Rodeo algorithm to iteratively select relevant variables and optimize bandwidths. It achieves a quasi-minimax convergence rate of $ O(n^{-2p/(2p+r)}) $, where $ r $ is the intrinsic dimension, while maintaining computational efficiency through early variable selection and sparsity-aware bandwidth tuning.

ABSTRACT

In this paper, we consider the problem of estimating a conditional density in moderately large dimensions. Much more informative than regression functions, conditional densities are of main interest in recent methods, particularly in the Bayesian framework (studying the posterior distribution, finding its modes...). Considering a recently studied family of kernel estimators, we select a pointwise multivariate bandwidth by revisiting the greedy algorithm Rodeo (Regularisation Of Derivative Expectation Operator). The method addresses several issues: being greedy and computationally efficient by an iterative procedure, avoiding the curse of high dimensionality under some suitably defined sparsity conditions by early variable selection during the procedure, converging at a quasi-optimal minimax rate.

Motivation & Objective

• To address the curse of dimensionality in conditional density estimation by exploiting sparsity in high-dimensional data.
• To develop a computationally efficient method that maintains fast convergence rates even when the number of relevant variables is small.
• To enable pointwise conditional density estimation in moderately large dimensions ($ d \gtrsim 3 $) where traditional kernel methods become intractable.
• To achieve minimax-optimal or near-minimax convergence rates under sparsity assumptions, specifically $ O(n^{-2p/(2p+r)}) $, where $ r $ is the number of relevant components.
• To integrate variable selection early in the estimation process, avoiding full-dimensional computation on irrelevant covariates.

Proposed method

• The method employs a revised version of the Rodeo (Regularisation Of Derivative Expectation Operator) greedy algorithm to iteratively select variables and tune bandwidths in a multivariate kernel density estimator.
• It uses a pointwise bandwidth selection strategy that adaptively identifies and retains only the most relevant components of the covariate vector $ X $, reducing dimensionality early in the process.
• The approach combines kernel density estimation with a sparsity-inducing penalty via iterative refinement, leveraging the derivative expectation operator to detect significant variables.
• Theoretical analysis relies on concentration inequalities and moment conditions to control estimation error, particularly bounding the difference between true and estimated marginal densities.
• The method ensures convergence under $ p $-regularity conditions and uses a dual-tree speed-up approximation to maintain computational tractability in moderate dimensions.
• It establishes oracle-type inequalities by controlling the bias-variance trade-off through adaptive bandwidth selection and sparsity detection.

Experimental results

Research questions

• RQ1 Can a nonparametric conditional density estimator achieve near-minimax rates in high-dimensional settings when only a small subset of covariates influence the response?
• RQ2 How can variable selection be integrated early in the bandwidth selection process to reduce computational cost without sacrificing estimation accuracy?
• RQ3 Is it possible to maintain computational efficiency in moderately high dimensions ($ d \gtrsim 3 $) while achieving optimal or near-optimal convergence rates?
• RQ4 To what extent does the method outperform standard kernel bandwidth selection (e.g., cross-validation) in terms of speed and accuracy under sparsity?
• RQ5 Can the Rodeo algorithm be adapted to conditional density estimation to ensure both consistency and fast convergence under sparsity?

Key findings

• The proposed method achieves a convergence rate of $ O(n^{-2p/(2p+r)}) $, which is nearly optimal under the minimax framework, where $ r $ is the number of relevant components influencing the conditional density.
• The method is computationally efficient due to early variable selection, avoiding full-dimensional bandwidth optimization and reducing runtime significantly.
• It maintains consistency and achieves the optimal rate even when the true dimensionality $ d $ is large, provided the intrinsic dimension $ r \ll d $.
• Theoretical guarantees are established via concentration bounds and moment conditions, showing that the estimation error is controlled with high probability.
• The method outperforms standard kernel bandwidth selection techniques (e.g., cross-validation) in terms of computational scalability in moderately high dimensions.
• Theoretical analysis confirms that the estimator achieves an oracle inequality, meaning it performs as well as if the true sparse structure were known in advance.