[Paper Review] GenDICE: Generalized Offline Estimation of Stationary Values

Ruiyi Zhang, Bo Dai+2 more|arXiv (Cornell University)|Feb 21, 2020

Reinforcement Learning in RoboticsComputer Science48 references50 citations

TL;DR

GenDICE is a consistent off-line estimator for stationary-value quantities using a stationary distribution correction ratio, applicable to off-policy evaluation and off-line PageRank.

ABSTRACT

An important problem that arises in reinforcement learning and Monte Carlo methods is estimating quantities defined by the stationary distribution of a Markov chain. In many real-world applications, access to the underlying transition operator is limited to a fixed set of data that has already been collected, without additional interaction with the environment being available. We show that consistent estimation remains possible in this challenging scenario, and that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions, derived from fundamental properties of the stationary distribution, and exploiting constraint reformulations based on variational divergence minimization. The resulting algorithm, GenDICE, is straightforward and effective. We prove its consistency under general conditions, provide an error analysis, and demonstrate strong empirical performance on benchmark problems, including off-line PageRank and off-policy policy evaluation.

Motivation & Objective

• Formalize the problem of off-line estimation of stationary quantities from fixed transition samples.
• Propose GenDICE, a stationary distribution correction estimator with a dual-embedding objective.
• Establish consistency and provide an error analysis for GenDICE.
• Demonstrate strong empirical performance on off-line PageRank and off-policy evaluation benchmarks.

Proposed method

• Formulate tau* = mu/pi correction as a density ratio tau = mu/p and derive a minimization of a D(T_gamma,mu0^p ◦ tau || p ◦ tau).
• Introduce a penalty-based feasible set to avoid degenerate solutions and prove that tau* = u/p solves the penalized problem for any lambda>0.
• Use a dual embedding with f-divergence to convert the problem into a min-max saddle-point objective that is tractable with samples.
• Specialize to chi-squared divergence for a practical J_chi^2 objective and parameterize tau and the dual function f with neural networks.
• Provide an unbiased gradient estimation for the saddle-point and implement a scalable SGD-based algorithm (GenDICE).
• Optionally extend to IPM-based divergences (e.g., MMD) and discuss trade-offs with kernel-based duals.

Experimental results

Research questions

• RQ1 How can stationary value estimation be performed from fixed off-line data without access to a transition sampler?
• RQ2 Can we define a correction ratio for the stationary distribution that yields consistent estimates in both undiscounted and discounted settings?
• RQ3 What is a practical, stable optimization objective to learn the ratio from samples, avoiding degenerate solutions?
• RQ4 How does GenDICE perform for off-line PageRank and off-policy evaluation compared to model-based or IS-based baselines?

Key findings

• GenDICE provides a consistent estimator for the stationary distribution correction under general conditions.
• By using a dual-embedding formulation with f-divergence, the method yields a convex objective in tau and remains stable for gamma in (0,1] including gamma=1.
• Theorem 1 shows the penalized objective with an E_p[tau]=1 constraint has tau* = u/p as the solution for any lambda>0, avoiding degenerate solutions.
• Chi-squared divergence renders the practical J_chi^2 objective amenable to neural network parameterizations for tau and the dual f.
• Empirical results show GenDICE outperforms model-based methods in tabular OPR and Taxi domain OPE settings, and remains competitive with or superior to DualDICE for discounted cases, with better stability near gamma=1.
• The method scales to continuous control tasks using neural nets and stochastic optimization.