[Paper Review] Kernel Bayes' rule

Kenji Fukumizu, Le Song+1 more|arXiv (Cornell University)|Sep 29, 2010

Gaussian Processes and Bayesian Inference35 references4 citations

TL;DR

This paper proposes a nonparametric kernel-based method for implementing Bayes' rule using reproducing kernel Hilbert spaces (RKHS), where probabilities are represented as RKHS means of empirical samples. The approach enables likelihood-free Bayesian inference and nonparametric filtering without explicit parametric models, with theoretical consistency rates and demonstrated applications in Bayesian computation and state-space modeling.

ABSTRACT

A nonparametric kernel-based method for realizing Bayes' rule is proposed, based on representations of probabilities in reproducing kernel Hilbert spaces. Probabilities are uniquely characterized by the mean of the canonical map to the RKHS. The prior and conditional probabilities are expressed in terms of RKHS functions of an empirical sample: no explicit parametric model is needed for these quantities. The posterior is likewise an RKHS mean of a weighted sample. The estimator for the expectation of a function of the posterior is derived, and rates of consistency are shown. Some representative applications of the kernel Bayes' rule are presented, including Baysian computation without likelihood and filtering with a nonparametric state-space model.

Motivation & Objective

• To develop a nonparametric method for Bayes' rule that avoids explicit parametric modeling of prior, likelihood, or posterior distributions.
• To represent probability measures as elements in a reproducing kernel Hilbert space (RKHS) via empirical samples.
• To enable Bayesian inference in complex models where likelihoods are intractable or unknown, such as in simulation-based inference.
• To provide theoretical consistency rates for the kernel Bayes' rule estimator in expectation over posterior functions.
• To demonstrate practical applications in likelihood-free Bayesian computation and nonparametric state-space filtering.

Proposed method

• Represent prior and conditional probabilities as RKHS elements using empirical samples via the canonical feature map.
• Express the posterior as the RKHS mean of a weighted empirical sample, with weights derived from kernelized likelihood ratios.
• Use kernel mean embeddings to nonparametrically represent all probabilistic quantities without assuming parametric forms.
• Derive an estimator for the posterior expectation of a function using weighted RKHS means, leveraging the representer theorem.
• Establish consistency rates for the estimator under mild regularity conditions, showing convergence to the true posterior expectation.
• Apply the method to likelihood-free inference by bypassing explicit likelihood computation and to filtering via nonparametric state-space models.

Experimental results

Research questions

• RQ1 Can Bayes' rule be implemented without explicit parametric models for prior, likelihood, or posterior?
• RQ2 How can probability distributions be nonparametrically represented and manipulated in a functional space like an RKHS?
• RQ3 What consistency rates can be achieved for kernel-based posterior estimators in expectation over functions?
• RQ4 Can kernel Bayes' rule enable effective inference in models with intractable likelihoods?
• RQ5 How can the method be applied to nonparametric filtering in state-space models without parametric assumptions?

Key findings

• The method enables nonparametric Bayesian inference without requiring explicit parametric forms for prior, likelihood, or posterior distributions.
• Probabilities are uniquely represented as RKHS means of empirical samples, allowing functional manipulation in a Hilbert space framework.
• The posterior is estimated as a weighted RKHS mean, with weights derived from kernelized likelihood ratios, avoiding explicit likelihood evaluation.
• The estimator for posterior expectations achieves consistency with theoretical convergence rates under mild regularity conditions.
• The approach supports likelihood-free Bayesian computation, allowing inference when likelihoods are intractable or unknown.
• Applications to nonparametric filtering in state-space models demonstrate the method’s utility in dynamic, complex systems without parametric assumptions.