[Paper Review] An Oracle Property of The Nadaraya-Watson Kernel Estimator for High Dimensional Nonparametric Regression

Daniel Conn, Gang Li|arXiv (Cornell University)|Nov 25, 2017

Statistical Methods and Inference3 citations

TL;DR

This paper establishes that the Nadaraya-Watson kernel estimator achieves an oracle property in high-dimensional nonparametric regression when the true regression function has a low-dimensional index structure. By allowing bandwidth matrices to diverge to infinity rather than converge to zero, and using K-fold cross-validation over positive semidefinite bandwidth matrices, the estimator attains a convergence rate dependent on the number of indices (m) rather than the full dimension (p), effectively mitigating the curse of dimensionality.

ABSTRACT

The celebrated Nadaraya-Watson kernel estimator is among the most studied method for nonparametric regression. A classical result is that its rate of convergence depends on the number of covariates and deteriorates quickly as the dimension grows, which underscores the "curse of dimensionality" and has limited its use in high dimensional settings. In this article, we show that when the true regression function is single or multi-index, the effects of the curse of dimensionality may be mitigated for the Nadaraya-Watson kernel estimator. Specifically, we prove that with $K$-fold cross-validation, the Nadaraya-Watson kernel estimator indexed by a positive semidefinite bandwidth matrix has an oracle property that its rate of convergence depends on the number of indices of the regression function rather than the number of covariates. Intuitively, this oracle property is a consequence of allowing the bandwidths to diverge to infinity as opposed to restricting them all to converge to zero at certain rates as done in previous theoretical studies. Our result provides a theoretical perspective for the use of kernel estimation in high dimensional nonparametric regression and other applications such as metric learning when a low rank structure is anticipated. Numerical illustrations are given through simulations and real data examples.

Motivation & Objective

• To address the curse of dimensionality in nonparametric regression, which degrades performance as the number of covariates (p) increases.
• To investigate whether the Nadaraya-Watson kernel estimator can maintain fast convergence rates in high-dimensional settings when the true regression function has a low-dimensional index structure.
• To establish theoretical justification for using continuous optimization over bandwidth matrices instead of discrete grids in kernel regression.
• To demonstrate that K-fold cross-validation over a bounded set of positive semidefinite bandwidth matrices yields optimal convergence rates when the regression function is single- or multi-index.

Proposed method

• Reformulates the Nadaraya-Watson estimator using a bandwidth matrix H that is positive semidefinite, allowing for low-rank structure in the regression function.
• Uses K-fold cross-validation to select the optimal bandwidth matrix H from a bounded subset of p×p positive semidefinite matrices.
• Applies an extended oracle inequality from Dudoit and van der Laan (2005) and Györfi et al. (2006) to bound the prediction risk of the cross-validated estimator.
• Derives convergence rates by relating the estimator to a lower-dimensional regression on the index space, using the fact that the bandwidth matrix can diverge to infinity.
• Employs bracketing entropy and Lipschitz continuity arguments to control the complexity of the parameter space and ensure uniform convergence.
• Uses a reparametrization where the kernel bandwidth is applied via H^{1/2}(X_i - x), enabling the estimator to exploit low-dimensional structure when H is rank-deficient.

Experimental results

Research questions

• RQ1 Can the Nadaraya-Watson kernel estimator achieve an oracle property in high-dimensional nonparametric regression when the true regression function is single- or multi-index?
• RQ2 Does allowing the bandwidth matrix to diverge to infinity instead of converging to zero improve the convergence rate in high-dimensional settings?
• RQ3 Can K-fold cross-validation over a continuous set of positive semidefinite bandwidth matrices yield optimal rates of convergence?
• RQ4 What is the rate of convergence of the cross-validated Nadaraya-Watson estimator when the true regression function depends on only m indices rather than p covariates?
• RQ5 Is there a theoretical justification for using gradient-based optimization over bandwidth matrices in kernel regression, rather than discrete grid search?

Key findings

• The Nadaraya-Watson kernel estimator achieves an oracle property: its rate of convergence depends on the number of indices (m) rather than the full dimension (p), provided the true regression function is single- or multi-index.
• The convergence rate is O(n^{-2/(m+2)}) under K-fold cross-validation, which is substantially faster than the classical O(n^{-2/(p+2)}) rate when m ≪ p.
• The bandwidth matrix H is allowed to diverge to infinity (i.e., not converge to zero), which is key to achieving the oracle property and avoiding the curse of dimensionality.
• The cross-validation criterion is minimized over a bounded subset of positive semidefinite matrices, enabling theoretical justification for continuous optimization techniques like gradient descent.
• The estimator's risk is bounded by a term that decays at rate O(log(n)^{m/(m+2)} n^{-2/(m+2)}), confirming the oracle property under mild regularity conditions.
• The result holds under the assumption that the outcome variable is almost surely bounded and the kernel function is Lipschitz continuous, with bracketing entropy conditions satisfied.