[Paper Review] Lasso in infinite dimension: application to variable selection in functional multivariate linear regression

Angelina Roche|arXiv (Cornell University)|Jan 1, 2018

Statistical Methods and Inference72 references5 citations

TL;DR

This paper proposes a functional Lasso method for variable selection in multivariate functional linear regression with infinite-dimensional covariates. It introduces two estimation strategies—Group-Lasso on the full Hilbert space and on finite-dimensional subspaces—with theoretical guarantees via oracle inequalities under fixed or random designs, and provides a coordinate descent algorithm for computation, validated on simulated and real data.

ABSTRACT

In more and more applications, a quantity of interest may depend on several covariates, with at least one of them infinite-dimensional (e.g. a curve). To select relevant covariate in this context, we propose an adaptation of the Lasso method. The criterion is based on classical Lasso inference under group sparsity (Yuan and Lin, 2006; Lounici et al., 2011). We give properties of the solution in our infinite-dimensional context. A sparsity-oracle inequality is shown and we propose a coordinate-wise descent algorithm, inspired by the glmnet algorithm (Friedman et al., 2007). A numerical study on simulated and experimental datasets illustrates the behavior of the method.

Motivation & Objective

• Address the challenge of variable selection in multivariate functional linear models where at least one covariate is infinite-dimensional.
• Develop a Lasso-based method adapted to functional data in high-dimensional or infinite-dimensional Hilbert spaces.
• Establish theoretical performance bounds (oracle inequalities) for variable selection in both fixed and random design settings.
• Provide a computationally feasible algorithm for estimating the functional Lasso solution in practice.
• Validate the method on both simulated and real-world datasets, including ozone prediction and nuclear safety applications.

Proposed method

• Define the multivariate functional linear model with a response Y and covariates in a product Hilbert space H = H₁ × ... × Hₚ, where at least one Hⱼ is infinite-dimensional.
• Propose two estimation methods: (1) Group-Lasso minimization over the full Hilbert space H, and (2) Group-Lasso minimization over a finite-dimensional subspace of H selected via penalized least squares.
• Use a coordinate descent algorithm to compute solutions for both estimation criteria, enabling practical implementation.
• Establish oracle inequalities for prediction risk under both fixed and random designs, showing the estimator adapts to sparsity in the true coefficient functions.
• Leverage concentration inequalities and spectral norm bounds to control the empirical and population Gram matrices, ensuring stability.
• Apply Bernstein’s inequality to bound the deviation of empirical inner products from their expectations, crucial for deriving high-probability bounds.

Experimental results

Research questions

• RQ1 Can the Lasso method be effectively extended to multivariate functional linear models with infinite-dimensional covariates?
• RQ2 How can variable selection be performed when some covariates are curves or functions in a Hilbert space?
• RQ3 What theoretical guarantees (e.g., prediction error bounds) can be derived for functional Lasso estimators under fixed or random designs?
• RQ4 How does the performance of the functional Lasso depend on the choice of finite-dimensional subspaces in the estimation process?
• RQ5 Can a coordinate descent algorithm be efficiently designed and implemented for functional Lasso in high-dimensional or infinite-dimensional settings?

Key findings

• The paper establishes oracle inequalities for both estimation methods, showing that the prediction risk of the functional Lasso estimator is bounded by the optimal risk plus a term that depends on the sparsity of the true coefficient functions.
• Theoretical analysis confirms that the method achieves optimal rates of convergence under sparsity assumptions, even when the design is random.
• The coordinate descent algorithm effectively computes the functional Lasso solution, enabling practical application to real datasets.
• The finite-dimensional subspace selection via penalized least squares leads to a stable and consistent estimation procedure with controlled approximation error.
• Numerical experiments on simulated and real data (e.g., ozone prediction, nuclear safety) demonstrate the method’s ability to correctly identify relevant functional and multivariate covariates.
• Theoretical bounds are derived using concentration inequalities and spectral norm control, with explicit dependence on the eigenstructure of the covariance operator and the design complexity.