[Paper Review] Approximations of the Reproducing Kernel Hilbert Space (RKHS) Embedding Method over Manifolds

Jia Guo, Sai Tej Paruchuri+1 more|arXiv (Cornell University)|Jul 13, 2020

Model Reduction and Neural Networks31 references4 citations

TL;DR

This paper establishes convergence guarantees and derives convergence rates for finite-dimensional approximations of the Reproducing Kernel Hilbert Space (RKHS) embedding method when estimating unknown functions in nonlinear ODEs. By embedding the RKHS into a Sobolev space and analyzing the approximation error via the fill distance of samples on a compact submanifold, the authors show that the error decays as a power of the number of samples, with rates dependent on kernel smoothness and Sobolev regularity.

ABSTRACT

The reproducing kernel Hilbert space (RKHS) embedding method is a recently introduced estimation approach that seeks to identify the unknown or uncertain function in the governing equations of a nonlinear set of ordinary differential equations (ODEs). While the original state estimate evolves in Euclidean space, the function estimate is constructed in an infinite-dimensional RKHS that must be approximated in practice. When a finite-dimensional approximation is constructed using a basis defined in terms of shifted kernel functions centered at the observations along a trajectory, the RKHS embedding method can be understood as a data-driven approach. This paper derives sufficient conditions that ensure that approximations of the unknown function converge in a Sobolev norm over a submanifold that supports the dynamics. Moreover, the rate of convergence for the finite-dimensional approximations is derived in terms of the fill distance of the samples in the embedded manifold. Numerical simulation of an example problem is carried out to illustrate the qualitative nature of convergence results derived in the paper.

Motivation & Objective

• To establish sufficient conditions for convergence of finite-dimensional RKHS approximations in Sobolev norms over a submanifold supporting the dynamics.
• To quantify the rate of convergence of the RKHS embedding estimator in terms of the fill distance of sampled data points on the manifold.
• To bridge the gap between theoretical infinite-dimensional RKHS estimation and practical finite-dimensional computation by analyzing approximation error in Sobolev spaces.
• To validate the theoretical convergence rates through numerical simulations on a nonlinear ODE with a one-dimensional invariant manifold.

Proposed method

• The RKHS embedding method is reformulated by embedding the RKHS into a Sobolev space $W^{ au,2}( dd)$, leveraging known embeddings for Matérn kernels.
• The approximation error is analyzed via the operator $I - \mathbf{P}_{\Omega_n}$, representing the projection error onto the finite-dimensional subspace $H_{\Omega_n}$ spanned by kernel functions centered at sampled points.
• Sobolev error bounds for radial basis function interpolation are applied to derive convergence rates in $W^{\mu,2}(\Omega)$-norm for the restricted function on the manifold.
• The fill distance $h_{\Omega_n,\Omega}$, measuring the maximal distance from any point on the manifold to the nearest sample, is used as the key parameter in the convergence rate analysis.
• The analysis assumes the trajectory is persistently excited (PE) on the manifold, ensuring that the sampling set $\Omega_n$ eventually fills the manifold densely.
• Numerical experiments use uniformly sampled points on a one-dimensional invariant manifold $\Omega = \{x \in \mathbb{R}^2 : \Phi(x) = -0.1\}$, with Matérn kernels of order $\nu = 3/2$ and $\nu = 5/2$.

Experimental results

Research questions

• RQ1 Under what conditions does the finite-dimensional approximation of the RKHS embedding estimator converge in Sobolev norm over a submanifold?
• RQ2 How does the rate of convergence depend on the fill distance of the sampling points on the manifold?
• RQ3 What is the relationship between the smoothness of the kernel (via $\nu$) and the resulting convergence rate in Sobolev and continuous norms?
• RQ4 Can the theoretical convergence rates be validated numerically using a nonlinear ODE with a known invariant manifold?
• RQ5 How does the choice of kernel (e.g., Matérn) affect the embedding of the RKHS into Sobolev spaces and the resulting approximation error?

Key findings

• The finite-dimensional approximation of the RKHS embedding estimator converges in the $W^{\mu,2}(\Omega)$-norm as the fill distance $h_{\Omega_n,\Omega}$ decreases.
• The convergence rate is bounded by $\|f - \hat{f}_n\|_{W^{\mu,2}(\Omega)} \sim N^{-(s - \mu)}$, where $N$ is the number of samples and $s$ depends on the kernel smoothness parameter $\nu$.
• For $\nu = 3/2$, the convergence rate in the $C(\Omega)$-norm is at least $O(N^{-1})$, and for $\nu = 5/2$, it improves to $O(N^{-2})$.
• Numerical results confirm that the actual error curves lie below the theoretical slope bounds derived from the $N^{-(s - \mu)}$ rate, validating the theoretical analysis.
• The convergence rate depends on the Sobolev regularity $\tau$ of the RKHS and the embedding into $W^{\tau - 0.5,2}(\Omega)$, with $s < 2\nu - 1.5$.
• The assumption that $h_{\Omega_n,\Omega}$ is below a threshold explains the flat error curve for $N \leq 30$, indicating a pre-asymptotic regime before the theoretical rate takes effect.