[Paper Review] Nonlinear function-on-function regression by RKHS

Peijun Sang, Bing Li|arXiv (Cornell University)|Jul 17, 2022

Statistical Methods and Inference4 citations

TL;DR

This paper proposes a nonlinear function-on-function regression model using nested reproducing kernel Hilbert spaces (RKHS) to capture complex, nonlinear relationships between functional covariates and functional responses. By constructing a second-layer RKHS on the first-layer Hilbert space of the covariate, the method enables flexible nonlinear modeling and establishes convergence rates and weak convergence for the predicted response under mild regularity conditions.

ABSTRACT

We propose a nonlinear function-on-function regression model where both the covariate and the response are random functions. The nonlinear regression is carried out in two steps: we first construct Hilbert spaces to accommodate the functional covariate and the functional response, and then build a second-layer Hilbert space for the covariate to capture nonlinearity. The second-layer space is assumed to be a reproducing kernel Hilbert space, which is generated by a positive definite kernel determined by the inner product of the first-layer Hilbert space for $X$--this structure is known as the nested Hilbert spaces. We develop estimation procedures to implement the proposed method, which allows the functional data to be observed at different time points for different subjects. Furthermore, we establish the convergence rate of our estimator as well as the weak convergence of the predicted response in the Hilbert space. Numerical studies including both simulations and a data application are conducted to investigate the performance of our estimator in finite sample.

Motivation & Objective

• To develop a nonlinear regression framework for functional data where both covariate and response are random functions.
• To address the lack of flexible, nonparametric methods for function-on-function regression in functional data analysis.
• To allow for irregularly spaced and sparse functional observations in the estimation procedure.
• To establish theoretical convergence rates and weak convergence properties for the proposed estimator.

Proposed method

• The method constructs a nested Hilbert space structure: the functional covariate $X$ resides in a Hilbert space $\mathcal{H}_X$, and a second-layer RKHS is built on $\mathcal{H}_X$ using a positive definite kernel derived from the inner product of $\mathcal{H}_X$.
• The regression operator maps from the second-layer RKHS to the response Hilbert space $\mathcal{H}_Y$, enabling nonlinear functional relationships.
• Estimation is performed using Tikhonov regularization to ensure stability and convergence, leveraging the representer theorem in the RKHS framework.
• The approach accommodates irregularly spaced and sparse functional data by using observed data points directly in the kernel-based estimation.
• The method relies on empirical risk minimization with a penalty term to control the complexity of the estimated operator.
• Weak convergence of the predicted response is established under mild regularity conditions, ensuring asymptotic validity of inference.

Experimental results

Research questions

• RQ1 Can a nonlinear function-on-function regression model be constructed to flexibly model complex dependencies between random functions?
• RQ2 How can the nested Hilbert space structure with RKHS enable nonlinearity while maintaining theoretical tractability?
• RQ3 What convergence rates can be achieved for the estimated regression operator under general conditions?
• RQ4 How does the method perform when functional data are observed at irregular or sparse time points?
• RQ5 Is the predicted response weakly convergent in the Hilbert space under the proposed estimation procedure?

Key findings

• The proposed estimator achieves a convergence rate that depends on the smoothness of the underlying regression operator and the eigenstructure of the covariance operators.
• The weak convergence of the predicted response in $\mathcal{H}_Y$ is established under mild moment and regularity conditions.
• The method maintains good predictive performance even when functional data are sparsely observed, as demonstrated in numerical studies.
• The theoretical analysis confirms asymptotic consistency of the estimator under regularity conditions on the covariate and response processes.
• The use of Tikhonov regularization ensures numerical stability and enables the application of the representer theorem in the RKHS framework.
• The nested Hilbert space structure allows for a nonparametric, flexible modeling of nonlinear functional relationships without assuming a parametric form.