[Paper Review] On Convergence of Emphatic Temporal-Difference Learning

Huizhen Yu|arXiv (Cornell University)|Jun 8, 2015

Reinforcement Learning in Robotics25 references15 citations

TL;DR

This paper presents the first convergence proofs for two emphatic temporal-difference learning algorithms, ETD(λ) and ELSTD(λ), in off-policy reinforcement learning with linear function approximation. It establishes L¹ convergence for ELSTD(λ) and almost sure convergence of value function estimates using a single infinite trajectory, under general off-policy conditions, using novel analytical techniques applicable beyond emphatic methods.

ABSTRACT

We consider emphatic temporal-difference learning algorithms for policy evaluation in discounted Markov decision processes with finite spaces. Such algorithms were recently proposed by Sutton, Mahmood, and White (2015) as an improved solution to the problem of divergence of off-policy temporal-difference learning with linear function approximation. We present in this paper the first convergence proofs for two emphatic algorithms, ETD($λ$) and ELSTD($λ$). We prove, under general off-policy conditions, the convergence in $L^1$ for ELSTD($λ$) iterates, and the almost sure convergence of the approximate value functions calculated by both algorithms using a single infinitely long trajectory. Our analysis involves new techniques with applications beyond emphatic algorithms leading, for example, to the first proof that standard TD($λ$) also converges under off-policy training for $λ$ sufficiently large.

Motivation & Objective

• To establish theoretical convergence guarantees for emphatic temporal-difference learning algorithms in off-policy settings.
• To address the known issue of divergence in off-policy TD learning with linear function approximation.
• To provide rigorous mathematical analysis of ETD(λ) and ELSTD(λ) under general off-policy data generation.
• To develop new analytical techniques applicable to broader classes of temporal-difference methods.
• To prove that standard TD(λ) also converges under off-policy training when λ is sufficiently large.

Proposed method

• Proposes ETD(λ) and ELSTD(λ) as off-policy temporal-difference algorithms using importance sampling and eligibility traces.
• Applies stochastic approximation theory to analyze convergence of the algorithms.
• Introduces a novel analysis framework involving weighted averages and empirical process techniques to handle off-policy data.
• Uses a single infinite trajectory to derive almost sure convergence of value function estimates.
• Corrects and refines earlier proofs in the literature, particularly regarding the convergence of Proposition C.1 and Footnote 17 in Appendix A.4.
• Establishes convergence under general off-policy conditions, including non-i.i.d. and non-uniformly distributed data.

Experimental results

Research questions

• RQ1 Does ETD(λ) converge under general off-policy conditions when using a single infinite trajectory?
• RQ2 Does ELSTD(λ) converge in L¹ under off-policy training with linear function approximation?
• RQ3 Can the theoretical analysis of emphatic TD learning be extended to prove convergence of standard TD(λ) under off-policy settings?
• RQ4 What novel analytical techniques are required to handle the non-i.i.d. and biased nature of off-policy data in TD learning?
• RQ5 How do corrections to earlier proofs affect the validity and generality of the convergence results?

Key findings

• ELSTD(λ) converges in L¹ under general off-policy conditions, establishing its theoretical reliability for value function estimation.
• The approximate value functions computed by both ETD(λ) and ELSTD(λ) converge almost surely along a single infinite trajectory.
• The analysis provides the first proof that standard TD(λ) also converges under off-policy training when λ is sufficiently large.
• The paper corrects and strengthens earlier proofs, particularly regarding the dependency of Proposition C.1 on the first part of Proposition C.2.
• The developed analytical techniques are generalizable and apply beyond emphatic algorithms, including to standard TD(λ) and other off-policy TD methods.
• The convergence results hold under minimal assumptions on the behavior policy and the underlying MDP, enhancing practical applicability.