[Paper Review] On Exploration, Exploitation and Learning in Adaptive Importance Sampling

Xiaoyu Lu, Tom Rainforth+3 more|arXiv (Cornell University)|Oct 31, 2018

Advanced Bandit Algorithms Research23 references4 citations

TL;DR

This paper proposes Daisee, a partition-based adaptive importance sampling algorithm that balances exploration and exploitation using bandit-inspired upper confidence bounds, achieving a cumulative pseudo-regret of $\mathcal{O}(\sqrt{T}(\log T)^{3/4})$. It further introduces HiDaisee, a hierarchical extension that adaptively learns the partitioning structure, improving sampling efficiency and empirical performance on complex targets.

ABSTRACT

We study adaptive importance sampling (AIS) as an online learning problem and argue for the importance of the trade-off between exploration and exploitation in this adaptation. Borrowing ideas from the bandits literature, we propose Daisee, a partition-based AIS algorithm. We further introduce a notion of regret for AIS and show that Daisee has $\mathcal{O}(\sqrt{T}(\log T)^{\frac{3}{4}})$ cumulative pseudo-regret, where $T$ is the number of iterations. We then extend Daisee to adaptively learn a hierarchical partitioning of the sample space for more efficient sampling and confirm the performance of both algorithms empirically.

Motivation & Objective

• To formalize adaptive importance sampling (AIS) as an online learning problem with explicit exploration-exploitation trade-offs.
• To develop a regret-minimizing AIS algorithm by borrowing from multi-armed bandit theory, particularly upper confidence bound (UCB) principles.
• To extend the method to learn hierarchical partitions of the sample space dynamically for improved efficiency and accuracy.
• To provide theoretical regret bounds for the proposed algorithm under KL divergence and $\alpha$-divergence measures.
• To empirically validate the performance of Daisee and HiDaisee on challenging target distributions, including high-dimensional and skewed densities.

Proposed method

• Daisee partitions the sample space into $K$ disjoint subsets and maintains a proposal distribution over these partitions.
• It uses a UCB-style optimism-in-the-face-of-uncertainty strategy to balance exploitation (based on estimated target density) and exploration (through confidence bounds).
• The algorithm updates proposal weights using a combination of empirical importance weights and exploration bonuses derived from the variance of estimates.
• The cumulative pseudo-regret is analyzed using the Kullback-Leibler (KL) divergence between the proposal and target distributions as the loss function.
• HiDaisee extends Daisee by learning the partitioning structure online via recursive splitting of nodes in a tree, guided by effective sample size (ESS) and a stopping criterion.
• The tree structure enables efficient sampling with $\mathcal{O}(\log K)$ cost per draw instead of $\mathcal{O}(K)$, by propagating estimates up the tree.

Experimental results

Research questions

• RQ1 Can the exploration-exploitation trade-off in adaptive importance sampling be formalized and optimized using online learning principles?
• RQ2 What theoretical regret bounds can be established for adaptive importance sampling under KL divergence loss?
• RQ3 How does hierarchical partitioning improve the efficiency and accuracy of adaptive importance sampling?
• RQ4 Can the algorithm dynamically adapt its partitioning to target density complexity without prior knowledge?
• RQ5 How does the performance of HiDaisee compare to existing methods in terms of convergence speed and estimation accuracy?

Key findings

• Daisee achieves a cumulative pseudo-regret of $\mathcal{O}(\sqrt{T} (\log T)^{3/4})$, indicating that per-iteration regret diminishes over time.
• The regret bound holds under mild assumptions and extends to $\alpha$-divergence measures, including variance of importance weights for $\alpha=2$.
• HiDaisee outperforms Daisee in final KL loss on a 1D target with discontinuous and non-flat regions, despite slower initial convergence.
• The hierarchical structure in HiDaisee self-optimizes, stopping splits in flat, low-density regions and continuing in high-density, non-uniform regions.
• On the 2D banana-shaped target, HiDaisee achieves lower squared error in marginal likelihood estimation than PI-MAIS, especially with higher sample counts and complex hierarchies.
• The number of partitions in HiDaisee converges to a stable, optimal structure, balancing estimation accuracy and computational cost.