[Paper Review] Positively Weighted Kernel Quadrature via Subsampling

Satoshi Hayakawa, Harald Oberhauser+1 more|arXiv (Cornell University)|Jul 20, 2021

Mathematical Approximation and Integration49 references4 citations

TL;DR

This paper proposes a novel method for constructing convex kernel quadrature rules using subsampling and spectral analysis of kernels, enabling efficient, theoretically grounded quadrature with positive weights. It achieves improved worst-case error rates—matching or outperforming state-of-the-art methods—using only i.i.d. samples and kernel evaluations, with strong empirical performance on synthetic and real-world datasets.

ABSTRACT

We study kernel quadrature rules with convex weights. Our approach combines the spectral properties of the kernel with recombination results about point measures. This results in effective algorithms that construct convex quadrature rules using only access to i.i.d. samples from the underlying measure and evaluation of the kernel and that result in a small worst-case error. In addition to our theoretical results and the benefits resulting from convex weights, our experiments indicate that this construction can compete with the optimal bounds in well-known examples.

Motivation & Objective

• To develop convex kernel quadrature rules with provably small worst-case error using only i.i.d. samples and kernel evaluations.
• To leverage spectral properties of kernels and recombination theorems (e.g., Carathéodory’s) to construct stable, positive-weight quadrature rules.
• To bridge the gap between theoretical convergence rates and empirical performance in kernel quadrature, especially for high-dimensional or complex domains.
• To provide efficient algorithms that avoid reliance on full Mercer decompositions or expectation computations, enhancing practical applicability.

Proposed method

• Uses recombination via Carathéodory’s theorem to construct convex quadrature rules from i.i.d. samples and kernel evaluations.
• Applies Mercer or Nyström approximation to the kernel integral operator to derive theoretical bounds on worst-case error.
• Employs convex optimization (CQP) to compute positive weights that minimize worst-case error under moment constraints.
• Introduces empirical variants (e.g., N. + empirical) that approximate theoretical bounds while improving computational efficiency.
• Uses randomized SVD and approximation heuristics (e.g., FNE) to accelerate computation without sacrificing accuracy.
• Employs a two-step measure reduction framework: first, approximate the target measure via subsampling; second, construct quadrature rules on the reduced measure.

Experimental results

Research questions

• RQ1 Can convex kernel quadrature rules be constructed efficiently using only i.i.d. samples and kernel evaluations, without requiring knowledge of Mercer decompositions?
• RQ2 What theoretical worst-case error rates can be achieved using spectral analysis and recombination theorems in convex kernel quadrature?
• RQ3 How do the proposed methods compare empirically to existing state-of-the-art methods like DPP, SBQ, or KT++ in terms of error and computational cost?
• RQ4 To what extent do empirical approximations (e.g., FNE) maintain accuracy while significantly reducing runtime?
• RQ5 How does the performance degrade when the empirical measure used for construction differs from the true measure, and what is the dominant error source in such settings?

Key findings

• The proposed Mercer-based method achieves a worst-case error bound of $ r_n $, matching the theoretical rate of optimal methods, though it requires knowledge of the Mercer decomposition.
• The Nyström-based method achieves a bound of $ n ilde{ ho}_n + r_{n+1} + rac{n}{ ilde{ ho}_n} $, which is less competitive theoretically but outperforms existing methods in experiments.
• The empirical variant (N. + empirical) achieves a bound of $ n ilde{ ho}_n + r_{n+1} + rac{n}{ ilde{ ho}_n} + rac{1}{N} $, with computational cost $ n ilde{ ho}_n N + n ilde{ ho}_n^2 + n^3 ext{log}(N/n) $, and performs competitively in practice.
• The FNE approximation method reduces runtime to ~2 seconds for $ n=160 $, while maintaining accuracy comparable to the full N. + empirical method.
• In experiments on real datasets (3D Road Network, Power Plant), the proposed methods (especially N. + empirical + opt and FNE + opt) outperform i.i.d. Bayesian quadrature and Nyström-based methods in error, with only moderate increases in runtime.
• When evaluated on a second empirical measure $ ilde{ u} $, the error of all methods eventually plateaus due to the MMD distance between $ u $ and $ ilde{ u} $, indicating that the method’s error is dominated by data approximation rather than quadrature construction.