[Paper Review] Discrete reproducing kernel Hilbert spaces: Sampling and distribution of Dirac-masses

Palle E. T. Jørgensen, Feng Tian|arXiv (Cornell University)|Jan 9, 2015

Mathematical Analysis and Transform Methods35 references22 citations

TL;DR

This paper investigates reproducing kernel Hilbert spaces (RKHSs) on infinite, discrete, countable sets, focusing on whether Dirac point-masses at each point are contained in the RKHS with finite norm. It establishes conditions under which Dirac masses are in the space, showing they are finite in discrete Brownian motion and resistor network models but not in binomial RKHSs, with applications to Gaussian free fields and Extreme Learning Machines.

ABSTRACT

We study reproducing kernels, and associated reproducing kernel Hilbert spaces (RKHSs) $\\mathscr{H}$ over infinite, discrete and countable sets $V$. In this setting we analyze in detail the distributions of the corresponding Dirac point-masses of $V$. Illustrations include certain models from neural networks: An Extreme Learning Machine (ELM) is a neural network-configuration in which a hidden layer of weights are randomly sampled, and where the object is then to compute resulting output. For RKHSs $\\mathscr{H}$ of functions defined on a prescribed countable infinite discrete set $V$, we characterize those which contain the Dirac masses $\\delta_{x}$ for all points $x$ in $V$. Further examples and applications where this question plays an important role are: (i) discrete Brownian motion-Hilbert spaces, i.e., discrete versions of the Cameron-Martin Hilbert space; (ii) energy-Hilbert spaces corresponding to graph-Laplacians where the set $V$ of vertices is then equipped with a resistance metric; and finally (iii) the study of Gaussian free fields.

Motivation & Objective

• To characterize which reproducing kernel Hilbert spaces (RKHSs) on infinite discrete sets contain Dirac point-masses with finite norm.
• To analyze the distribution and norm of Dirac masses in concrete models, including discrete Brownian motion, Brownian bridge, and binomial kernels.
• To establish connections between energy-Hilbert spaces from graph-Laplacians and RKHSs, particularly in infinite resistor networks.
• To prove universality of certain RKHSs derived from electrical networks, enabling uniform approximation of continuous functions.
• To apply results to extreme learning machines and Gaussian free fields, linking discrete analysis to machine learning and stochastic processes.

Proposed method

• Uses the theory of positive definite kernels on countable discrete sets $V$ to define associated RKHSs $\mathscr{H}(k)$ via the reproducing property $f(x) = \langle f, k_x \rangle_\mathscr{H}$.
• Applies the Stone-Weierstrass theorem to prove universality of RKHSs from resistor networks, under conditions on the conductance function $c$.
• Computes the $\mathscr{H}$-norm of Dirac masses $\delta_x$ using the resistance metric and energy forms derived from graph-Laplacians.
• Derives explicit formulas for $\|\delta_x\|_\mathscr{H}^2$ in terms of local conductance and path structure, e.g., $\|\delta_\alpha\|_\mathscr{H}^2 = \frac{2}{r(l(\alpha))} + \frac{1}{r(l(\alpha)-1)}$ for the binary tree.
• Establishes isomorphism between the RKHS of the Green's function of a graph-Laplacian and the energy-Hilbert space via the Dirichlet form.
• Uses metric completion of the resistance metric on vertex sets to analyze convergence and boundary behavior of point-masses.

Experimental results

Research questions

• RQ1 Under what conditions on a positive definite kernel $k$ on a countable discrete set $V$ is the Dirac mass $\delta_x$ in the associated RKHS $\mathscr{H}(k)$ with finite norm?
• RQ2 Are Dirac masses in the RKHS for discrete Brownian motion and Brownian bridge processes, and what is their norm?
• RQ3 Does the RKHS associated with an infinite resistor network (with conductance function $c$) contain all Dirac masses, and can it uniformly approximate continuous functions?
• RQ4 What is the explicit formula for the $\mathscr{H}$-norm of a Dirac mass $\delta_x$ in a resistor network or tree structure?
• RQ5 How does the resistance metric on the vertex set $V$ relate to the metric completion and the norm of Dirac masses in the RKHS?

Key findings

• The Dirac mass $\delta_x$ has finite $\mathscr{H}$-norm in discrete Brownian motion and resistor network models, as shown in Theorem 4.13 and Corollary 4.5.
• For the binary tree with conductance $c_{\alpha,\alpha t} = 1/r(l(\alpha))$, the norm of $\delta_\alpha$ is $\|\delta_\alpha\|_\mathscr{H}^2 = \frac{2}{r(l(\alpha))} + \frac{1}{r(l(\alpha)-1)}$, which diverges as $\alpha$ approaches the boundary.
• The RKHS from the Green's function of a graph-Laplacian on a discrete set is isometric to the energy-Hilbert space, and all Dirac masses are in the space.
• The RKHS associated with a resistor network is universal: it can uniformly approximate any continuous function on compact subsets of the metric completion of $V$.
• In the binomial RKHS, the Dirac masses $\delta_x$ do not have finite norm, as shown in Theorem 3.18.
• The resistance metric on the vertex set $V$ of an infinite resistor network satisfies $\widetilde{R^{(c)}}(\omega,\omega') = 2\sum_{n=l(\omega \cap \omega')}^{\infty} r(n)$, and this metric completion is isometric to the boundary of the tree.