[Paper Review] Stochastic Variance Reduction Methods for Policy Evaluation

Simon S. Du, Jianshu Chen+3 more|arXiv (Cornell University)|Feb 25, 2017

Reinforcement Learning in Robotics69 citations

TL;DR

The paper transforms empirical MSPBE into a convex-concave saddle-point problem and develops primal-dual and stochastic variance reduction algorithms (SVRG and SAGA) that converge linearly for policy evaluation with linear function approximation.

ABSTRACT

Policy evaluation is a crucial step in many reinforcement-learning procedures, which estimates a value function that predicts states' long-term value under a given policy. In this paper, we focus on policy evaluation with linear function approximation over a fixed dataset. We first transform the empirical policy evaluation problem into a (quadratic) convex-concave saddle point problem, and then present a primal-dual batch gradient method, as well as two stochastic variance reduction methods for solving the problem. These algorithms scale linearly in both sample size and feature dimension. Moreover, they achieve linear convergence even when the saddle-point problem has only strong concavity in the dual variables but no strong convexity in the primal variables. Numerical experiments on benchmark problems demonstrate the effectiveness of our methods.

Motivation & Objective

• Motivate efficient policy evaluation under a fixed dataset for reinforcement learning.
• Formulate MSPBE minimization as a convex-concave saddle-point problem with finite-sum structure.
• Develop scalable algorithms that have linear convergence and low per-iteration cost.
• Provide theoretical convergence guarantees under mild assumptions through SVRG and SAGA variants.
• Demonstrate effectiveness via experiments on benchmark RL problems.

Proposed method

• Transform EM-MSPBE into a convex-concave saddle-point problem using conjugate functions.
• Derive a primal-dual batch gradient method (PDBG) with linear convergence under full-rank coupling and positive definite C.
• Introduce SVRG and SAGA variants that exploit finite-sum structure for fast variance-reduced updates.
• Ensure per-iteration cost is Z(d^2) or better and scale linearly with sample size n and feature dimension d.
• Provide convergence guarantees showing linear rates even when the primal objective lacks strong convexity, relying on the bilinear coupling A and C being well-conditioned.
• Discuss extensions to off-policy learning and TD with eligibility traces.

Experimental results

Research questions

• RQ1 Can EM-MSPBE be reformulated into a finite-sum convex-concave saddle-point problem suitable for variance-reduction methods?
• RQ2 Do SVRG and SAGA achieve linear convergence in policy evaluation when the saddle-point problem has only dual strong concavity (rho may be zero) and a full-rank coupling matrix?
• RQ3 How do the proposed methods compare in computational cost and convergence to existing policy evaluation algorithms on standard RL datasets?
• RQ4 What are the implications for off-policy learning and eligibility traces within this saddle-point framework?

Key findings

• SVRG and SAGA achieve linear convergence for policy evaluation with EM-MSPBE, under Assumption 1 (full-rank  and positive definite Ĉ).
• The primal-dual formulation enables linear convergence even when only dual strong concavity is present (rho = 0) due to the full-rank bilinear coupling.
• Per-iteration cost scales linearly with dataset size n and feature dimension d, with efficient gradient-based updates.
• SVRG/SAGA outperform traditional gradient methods and LSTD in regimes with large n or large d, according to complexity discussions.
• The approach extends to off-policy learning and TD with eligibility traces without sacrificing convergence guarantees.