[Paper Review] Batch Stationary Distribution Estimation

Junfeng Wen, Bo Dai+2 more|arXiv (Cornell University)|Mar 2, 2020

Advanced Queuing Theory Analysis50 references12 citations

TL;DR

This paper proposes a variational power method (VPM) for estimating the stationary distribution of an ergodic Markov chain using a fixed batch of transition data, without access to the underlying process or the probe distribution. VPM consistently estimates the ratio between the stationary and probe distributions via a variational formulation, enabling accurate post-processing of data in queueing systems, SDEs, MCMC, and off-policy evaluation, with significant improvements over existing methods.

ABSTRACT

We consider the problem of approximating the stationary distribution of an ergodic Markov chain given a set of sampled transitions. Classical simulation-based approaches assume access to the underlying process so that trajectories of sufficient length can be gathered to approximate stationary sampling. Instead, we consider an alternative setting where a fixed set of transitions has been collected beforehand, by a separate, possibly unknown procedure. The goal is still to estimate properties of the stationary distribution, but without additional access to the underlying system. We propose a consistent estimator that is based on recovering a correction ratio function over the given data. In particular, we develop a variational power method (VPM) that provides provably consistent estimates under general conditions. In addition to unifying a number of existing approaches from different subfields, we also find that VPM yields significantly better estimates across a range of problems, including queueing, stochastic differential equations, post-processing MCMC, and off-policy evaluation.

Motivation & Objective

• To address the challenge of estimating stationary distribution properties when only a fixed batch of transitions is available, without access to the underlying process or probe distribution.
• To develop a consistent, model-free estimator that recovers the stationary distribution from pre-collected data using only transition pairs.
• To unify and improve upon existing approaches in queueing systems, stochastic differential equations, MCMC post-processing, and off-policy evaluation.
• To enable behavior-agnostic off-policy evaluation in reinforcement learning by estimating the stationary distribution of state-action pairs from behavior policy data.

Proposed method

• Proposes a variational reformulation of the power method to bypass the intractable functional operations in the original power iteration for stationary distribution estimation.
• Estimates the ratio function τ(x) ≈ μ(x)/p(x) between the stationary and probe distributions using a variational optimization objective.
• Uses a neural network or function approximator to represent the ratio function τ(x), trained via a variational objective that enforces the fixed-point condition of the Markov chain.
• Applies the learned ratio to reweight the empirical distribution of collected states, yielding a consistent estimate of the stationary distribution.
• Employs a two-step process: first, estimate the correction ratio; second, use it to compute expectations under the stationary measure.
• Demonstrates that VPM avoids the collapse issues of model-based methods by directly learning the ratio without modeling the transition kernel.

Experimental results

Research questions

• RQ1 Can a consistent estimator for the stationary distribution be constructed from a fixed batch of transitions without additional data collection or knowledge of the probe distribution?
• RQ2 How can the power method be adapted to the batch, agnostic setting where the transition operator is unknown and functional operations are intractable?
• RQ3 To what extent can the proposed variational power method improve estimation accuracy in queueing systems, SDEs, MCMC, and off-policy evaluation compared to existing methods?
• RQ4 Does the VPM approach remain robust and consistent when the probe distribution is unknown and the data is collected under arbitrary sampling mechanisms?
• RQ5 Can VPM achieve behavior-agnostic off-policy evaluation by estimating the stationary state-action distribution from behavior policy transitions alone?

Key findings

• VPM achieves significantly better stationary distribution estimates than model-based baselines and importance sampling methods across all tested domains, including queueing, SDEs, and MCMC.
• In MCMC post-processing, VPM reduces the Maximum Mean Discrepancy (MMD) between estimated and true samples by consistently improving the quality of resampled data across different MCMC steps.
• For off-policy evaluation, VPM outperforms all baselines—including behavior-agnostic methods like Liu et al. (2018) and weighted importance sampling—by achieving lower log MSE on the Taxi, Reacher, HalfCheetah, and Ant environments.
• The method avoids the data collapse problem common in model-based approaches by learning the ratio function directly, preserving diversity and accuracy in the estimated stationary distribution.
• Empirical results show consistent improvement in MMD and MSE across multiple datasets and settings, demonstrating the robustness and generalization of VPM.
• Theoretical analysis confirms that VPM provides provably consistent estimates of the stationary distribution under general conditions, even without knowledge of the probe distribution or transition kernel.