[Paper Review] Nonparametric sparsity and regularization

Lorenzo Rosasco, Silvia Villa+3 more|arXiv (Cornell University)|Aug 13, 2012

Sparse and Compressive Sensing Techniques63 references59 citations

TL;DR

This paper introduces a novel nonparametric sparsity framework for variable selection in nonlinear supervised learning by measuring variable importance through partial derivatives in a reproducing kernel Hilbert space (RKHS). It proposes a data-driven regularization scheme using proximal methods to solve the resulting non-differentiable convex optimization problem, achieving consistent selection of relevant variables and superior empirical performance over state-of-the-art methods.

ABSTRACT

In this work we are interested in the problems of supervised learning and variable selection when the input-output dependence is described by a nonlinear function depending on a few variables. Our goal is to consider a sparse nonparametric model, hence avoiding linear or additive models. The key idea is to measure the importance of each variable in the model by making use of partial derivatives. Based on this intuition we propose a new notion of nonparametric sparsity and a corresponding least squares regularization scheme. Using concepts and results from the theory of reproducing kernel Hilbert spaces and proximal methods, we show that the proposed learning algorithm corresponds to a minimization problem which can be provably solved by an iterative procedure. The consistency properties of the obtained estimator are studied both in terms of prediction and selection performance. An extensive empirical analysis shows that the proposed method performs favorably with respect to the state-of-the-art methods.

Motivation & Objective

• To address variable selection in high-dimensional nonlinear regression where the true function depends on only a few relevant variables.
• To develop a nonparametric sparsity measure based on partial derivatives rather than linear or additive assumptions.
• To design a stable, computationally feasible regularization scheme using RKHS theory and proximal optimization.
• To establish theoretical consistency of the estimator in both prediction and variable selection.
• To empirically validate the method against state-of-the-art approaches on synthetic and real-world datasets.

Proposed method

• Proposes a new notion of nonparametric sparsity based on the L2 norm of partial derivatives of the function in the RKHS.
• Defines a data-dependent regularizer using empirical estimates of partial derivatives at training points.
• Uses the RKHS framework to ensure boundedness and stability of derivative estimators via the representer theorem and kernel-based function representation.
• Develops an iterative forward-backward splitting algorithm to solve the non-smooth convex optimization problem arising from the regularized least squares objective.
• Applies proximal methods to handle the non-differentiability of the regularization term, enabling convergence guarantees.
• Employs concentration inequalities and RKHS norm control to derive finite-sample bounds on the estimation error of derivative norms.

Experimental results

Research questions

• RQ1 Can partial derivatives in a RKHS provide a reliable and stable measure of variable importance in nonlinear, non-additive models?
• RQ2 Can a regularization scheme based on partial derivative norms consistently identify the true set of relevant variables in high-dimensional nonlinear regression?
• RQ3 How does the proposed method compare in prediction accuracy and variable selection performance to existing L1-regularized or additive model-based approaches?
• RQ4 What are the theoretical consistency properties of the estimator in terms of prediction error and support recovery?
• RQ5 Can the iterative proximal algorithm reliably converge to a solution with provable convergence rates?

Key findings

• The proposed estimator achieves consistent variable selection: the probability that all relevant variables are recovered approaches one as sample size increases.
• The method demonstrates strong empirical performance, outperforming state-of-the-art methods in both prediction accuracy and sparsity recovery across multiple datasets.
• Theoretical analysis shows that the estimated derivative norm converges to the true derivative norm in probability, with convergence rates depending on sample size and regularization parameters.
• The algorithm converges under mild assumptions, with convergence rates established via proximal method theory and RKHS concentration bounds.
• The consistency of the selection procedure is proven under conditions on the regularization parameter τn, which must satisfy limn→∞ τn = 0 and limn→∞ a(n, τn) = 0.
• The method is robust to high-dimensional input, as shown by theoretical bounds that scale favorably with dimension d and sample size n.