[Paper Review] Efficient and principled score estimation with Nyström kernel exponential families

Danica J. Sutherland, Heiko Strathmann+2 more|arXiv (Cornell University)|May 23, 2017

Statistical Methods and Inference6 references3 citations

TL;DR

This paper proposes a computationally efficient and statistically principled method for learning infinite-dimensional exponential family models using Nyström approximation to the score matching solution. By leveraging low-rank kernel approximation, the approach achieves the same convergence rates as the full-rank solution in Fisher, $L^r$, and Hellinger distances, with guaranteed cost and storage reduction, while outperforming heuristic approximations and autoencoder-based score estimators in accuracy, speed, and interpretability.

ABSTRACT

We propose a fast method with statistical guarantees for learning an exponential family density model where the natural parameter is in a reproducing kernel Hilbert space, and may be infinite-dimensional. The model is learned by fitting the derivative of the log density, the score, thus avoiding the need to compute a normalization constant. Our approach improves the computational efficiency of an earlier solution by using a low-rank, Nyström-like solution. The new solution retains the consistency and convergence rates of the full-rank solution (exactly in Fisher distance, and nearly in other distances), with guarantees on the degree of cost and storage reduction. We evaluate the method in experiments on density estimation and in the construction of an adaptive Hamiltonian Monte Carlo sampler. Compared to an existing score learning approach using a denoising autoencoder, our estimator is empirically more data-efficient when estimating the score, runs faster, and has fewer parameters (which can be tuned in a principled and interpretable way), in addition to providing statistical guarantees.

Motivation & Objective

• To develop a fast, scalable method for learning infinite-dimensional exponential family models without requiring normalization constants.
• To address the high computational cost ($\mathcal{O}(n^3d^3)$) and memory ($\mathcal{O}(n^2d^2)$) of full-rank kernel exponential family estimation.
• To provide theoretical guarantees on convergence and approximation error for a low-rank Nyström-based solution.
• To improve data efficiency, speed, and interpretability over existing score learning methods like denoising autoencoders.
• To enable practical deployment of infinite exponential families in applications such as adaptive Hamiltonian Monte Carlo sampling.

Proposed method

• The method uses a Nyström approximation to reduce the rank of the kernel matrix in the score matching solution, significantly lowering computational and storage costs.
• It applies a Tikhonov-regularized solution to the score matching objective, avoiding the need to compute the intractable normalization constant.
• The algorithm selects a subset of $m$ Nyström points to approximate the full kernel matrix, with $m \sim n^{1/3}\log n$ to $n^{1/2}\log n$ for convergence.
• Theoretical guarantees are derived using recent techniques from least-squares regression, directly controlling generalization error via the Nyström basis.
• The method maintains consistency and convergence rates matching the full-rank solution in Fisher distance and nearly matching in other distances.
• The approach is implemented via a linear system defined by first and second derivatives of the RKHS kernel at sample points, enabling efficient optimization.

Experimental results

Research questions

• RQ1 Can a low-rank Nyström approximation preserve the statistical consistency and convergence rates of the full-rank kernel exponential family solution?
• RQ2 What is the minimal number of Nyström points $m$ required to achieve convergence as $n \to \infty$?
• RQ3 How does the approximation error scale with respect to the number of training samples $n$ and Nyström points $m$?
• RQ4 Can the method outperform heuristic approximations (e.g., random Fourier features) in both accuracy and computational efficiency?
• RQ5 How does the proposed score estimator compare to deep learning-based alternatives like denoising autoencoders in data efficiency and interpretability?

Key findings

• The method achieves the same convergence rates as the full-rank solution in Fisher distance, with $m \sim n^{1/3}\log n$ to $n^{1/2}\log n$ Nyström points.
• Convergence in $L^r$, Hellinger, and KL distances matches or is nearly as fast as the full solution, depending on density smoothness.
• The approach is empirically more data-efficient than the denoising autoencoder score estimator, with fewer tunable parameters and better interpretability.
• It runs faster than the full-rank solution and performs as well as previous heuristic approximations at a fraction of the computational cost.
• In adaptive Hamiltonian Monte Carlo, the learned score leads to higher acceptance rates, demonstrating practical utility.
• Theoretical bounds on generalization error are derived directly from the Nyström basis, avoiding indirect reconstruction error arguments.