[Paper Review] Importance Sampling: Computational Complexity and Intrinsic Dimension

Sergios Agapiou, Omiros Papaspiliopoulos+2 more|arXiv (Cornell University)|Nov 19, 2015

Gaussian Processes and Bayesian Inference3 references19 citations

TL;DR

This paper introduces a unified framework for analyzing the computational complexity of importance sampling by linking it to the intrinsic dimension of the underlying measures. It demonstrates that the difference between the proposal and target measures—quantified via intrinsic dimension—determines sampling efficiency, offering practitioners a principled way to assess when importance sampling will succeed or fail.

ABSTRACT

The basic idea of importance sampling is to use independent samples from one measure in order to approximate expectations with respect to another measure. Understanding how many samples are needed is key to understanding the computational complexity of the method, and hence to understanding when it will be effective and when it will not. It is intuitive that the size of the difference between the measure which is sampled, and the measure against which expectations are to be computed, is key to the computational complexity. An implicit challenge in many of the published works in this area is to find useful quantities which measure this difference in terms of parameters which are pertinent for the practitioner. The subject has attracted substantial interest recently from within a variety of communities. The objective of this paper is to overview and unify the resulting literature in the area by creating an overarching framework. The general setting is studied in some detail, followed by deeper development in the context of Bayesian inverse problems and filtering.

Motivation & Objective

• To unify disparate literature on importance sampling by identifying a common theoretical foundation.
• To quantify the computational complexity of importance sampling in terms of measurable, practitioner-relevant parameters.
• To establish intrinsic dimension as a key metric for predicting sampling efficiency.
• To apply the framework to Bayesian inverse problems and filtering, where importance sampling is widely used.
• To provide a theoretical basis for understanding when importance sampling becomes computationally infeasible.

Proposed method

• Proposes a general theoretical framework to analyze importance sampling in terms of the divergence between proposal and target measures.
• Introduces intrinsic dimension as a geometric measure of the difference between probability measures.
• Uses the Kullback-Leibler divergence and related information-theoretic quantities to characterize sampling complexity.
• Applies the framework to Bayesian inverse problems by analyzing the geometry of posterior distributions.
• Extends the analysis to filtering problems, particularly sequential importance sampling, using the same intrinsic dimension metric.
• Derives conditions under which the effective sample size degrades exponentially with dimension, linking this to intrinsic dimension.

Experimental results

Research questions

• RQ1 How can the computational complexity of importance sampling be quantified in terms of measurable, interpretable parameters?
• RQ2 What role does the intrinsic dimension of the measure difference play in determining sampling efficiency?
• RQ3 In what settings—particularly Bayesian inverse problems and filtering—does importance sampling become intractable?
• RQ4 How does the divergence between proposal and target measures relate to the number of samples required for accurate estimation?
• RQ5 Can intrinsic dimension serve as a reliable predictor of importance sampling performance across different applications?

Key findings

• The computational complexity of importance sampling is fundamentally governed by the intrinsic dimension of the difference between the proposal and target measures.
• When the intrinsic dimension is high, the number of samples required for accurate estimation grows exponentially, making importance sampling infeasible.
• The framework provides a principled way to assess when importance sampling will fail due to high intrinsic dimension, even with well-chosen proposals.
• In Bayesian inverse problems, the intrinsic dimension of the posterior relative to the prior determines the scalability of importance sampling.
• For filtering problems, the intrinsic dimension of the likelihood and prior combination dictates the effective sample size over time.
• The results show that traditional measures like the Kullback-Leibler divergence alone are insufficient to predict performance; intrinsic dimension provides a more informative metric.