[Paper Review] Online Off-policy Prediction

Sina Ghiassian|arXiv (Cornell University)|Nov 6, 2018

Advanced Bandit Algorithms Research26 references14 citations

TL;DR

This paper presents a comprehensive empirical study of online off-policy temporal difference learning algorithms for value function prediction, evaluating methods like Emphatic-TD, Gradient-TD, and V-trace in two challenging microworlds. It finds that ETD(λ) offers superior asymptotic performance and robustness to hyperparameters, while correcting the entire TD-error with importance sampling consistently outperforms partial correction.

ABSTRACT

In this dissertation, we study online off-policy temporal-difference learning algorithms, a class of reinforcement learning algorithms that can learn predictions in an efficient and scalable manner. The contributions of this dissertation are one of the two kinds: (1) empirically studying existing off-policy learning algorithms, or, (2) exploring new algorithmic ideas. In reinforcement learning, it is not uncommon for an agent to learn about one policy, called the target policy, while behaving using a different policy, called the behavior policy. When the target and behavior policies are different, learning is `off' the policy in the sense that the data is from a different source than the target policy. Our first contribution is a novel Gradient-TD algorithm called TDRC. Gradient-TD algorithms are one of the most important families of off-policy learning algorithms due to their favorable convergence guarantees. Previous research showed that Gradient-TD algorithms learn slower than some other unsound algorithms such as Off-policy TD (Maei, 2011). In addition, Gradient-TD algorithms are not as easy to use as Off-policy TD because they have two learned weight vectors and two step-size parameters. Our contribution is a new algorithm called temporal-difference learning with regularized corrections (TDRC), that behaves as well as Off-policy TD, when Off-policy TD performs well but is sound in cases where Off-policy TD diverges. TDRC provides a standard way of setting the second step-size parameter and is easier to use than other Gradient-TD algorithms. We empirically investigate the performance of TDRC across a range of problems, for both prediction and control, and for both linear and non-linear function approximation, and show that it consistently outperforms other Gradient-TD algorithms. We derive the control variant of TDRC and show that it could be a better alternative to existing algorithms for solving complex control problems with neural networks. One of the important contributions of this thesis is a comprehensive empirical study of off-policy prediction learning algorithms. The study is one of the most thorough studies of off-policy prediction learning algorithms to date in terms of the number of algorithms it includes and is unique in that the performance of algorithms with respect to each of the algorithms parameters is assessed individually. We present empirical results with eleven prominent off-policy learning algorithms that use linear function approximation: five Gradient-TD methods, two Emphatic-TD methods, Off-policy TD, V-trace, Tree Backup, and ABQ. We assess the performance of the algorithms in terms of their learning speed, asymptotic error, and sensitivity to the step-size and bootstrapping parameters on a simple problem, called the Collision task, that has eight states and two actions. On the Collision task, we found that the two Emphatic-TD algorithms learned the fastest, reached the lowest errors, and were robust to parameter settings. V-trace, Tree Backup, and ABQ were no faster and had higher asymptotic error than other algorithms. Another one of the main contributions of this thesis is the first empirical study of off-policy prediction learning algorithms with a focus on one of the most important challenges in off-policy learning: slow learning. When learning off-policy, importance sampling ratios are used to correct for the differences between the target and behavior policies. These ratios can be large and of high variance. The step-size parameter should be set small to control this high variance, which in turn slows down learning. In this study, we found that the algorithms' performance is highly affected by the variance induced by the importance sampling ratios. Data shows that algorithms that adapt the bootstrapping parameter during learning, such as V-trace are less affected by the high variance than the other algorithms. We observed that Emphatic TD tends to have lower asymptotic error than other algorithms, but exhibits high variance and does not learn well when the product of importance sampling ratios is large. The final contribution of this thesis is a step-size adaptation algorithm, called Step-size Ratchet, that when combined with Emphatic TD, significantly improves its learning speed. The proposed algorithm keeps the step-size parameter as large as possible and only ratchets it down when necessary. We establish the effectiveness of the combination of Emphatic TD and Step-size Ratchet by comparing it with the combination of Emphatic TD and other step-size adaptation algorithms on tasks where learning fast is challenging. (Abridged abstract)

Motivation & Objective

• To close the empirical gap in understanding the relative performance of modern online off-policy prediction algorithms.
• To evaluate the impact of importance sampling placement—correcting the full TD-error versus only the value estimate—on learning stability and performance.
• To assess the sensitivity, convergence rate, and asymptotic performance of key algorithms under varying hyperparameters and environmental variance.
• To clarify the practical advantages and limitations of methods like Emphatic-TD, Gradient-TD, V-trace, and Tree Backup in fixed-basis function settings.

Proposed method

• The authors evaluate 14 online off-policy prediction algorithms, including Emphatic-TD, Gradient-TD, V-trace, Tree Backup, and ABTD(ζ), across two complex microworlds with fixed-basis function representations.
• They systematically vary hyperparameters such as stepsize, λ, and β to assess sensitivity and convergence behavior.
• Importance sampling corrections are applied either to the entire TD-error or only to the value function estimate, and the impact on performance is empirically compared.
• The study uses area-under-the-curve (AUC) optimization to tune hyperparameters and assesses learning speed, variance, and final performance.
• The analysis includes both low-variance and high-variance settings for the importance sampling ratio to evaluate robustness.
• The authors compare methods with and without explicit prior corrections (e.g., V-trace) and with different eligibility trace mechanisms.

Experimental results

Research questions

• RQ1 How do different off-policy prediction algorithms compare in terms of asymptotic performance, convergence speed, and sensitivity to hyperparameters?
• RQ2 Does correcting the entire TD-error term with importance sampling yield better performance than correcting only the value function estimate?
• RQ3 How do methods like ETD(λ) and V-trace perform under high-variance importance sampling ratios and long eligibility traces?
• RQ4 What is the role of prior corrections (e.g., in V-trace and Tree Backup) in stabilizing learning and improving performance?
• RQ5 Do hybrid or proximal variants of Gradient-TD offer meaningful improvements over standard Gradient-TD in practice?

Key findings

• Emphatic-TD(λ) achieves better asymptotic performance than Gradient-TD across all tested settings, especially for smaller λ values.
• Correcting the entire TD-error term with importance sampling consistently outperforms or matches partial correction (only v(S_t)) and is therefore recommended.
• ETD(λ) is significantly less sensitive to the λ hyperparameter than other methods and performs well even in high-variance settings.
• V-trace(λ) and Tree Backup(λ) do not dominate each other; performance depends on the environment, with V-trace often excelling in high-variance scenarios.
• ABTD(ζ) performs comparably to the best methods and sometimes outperforms both V-trace and Tree Backup, particularly in high-variance tasks.
• ETD(λ,β) does not provide a clear advantage over ETD(λ), and its performance is best when β ≈ γ, reducing it to ETD(λ).