[Paper Review] Off-policy Multi-step Q-learning

Gabriel Kalweit, Maria Huegle+1 more|arXiv (Cornell University)|Sep 25, 2019

Reinforcement Learning in Robotics11 references4 citations

TL;DR

This paper proposes Composite Q-learning, an off-policy multi-step temporal-difference method using truncated and shifted Q-functions to model short- and long-term returns, respectively. It improves data efficiency in deep reinforcement learning, outperforming TD3 and state-of-the-art off-policy methods on simulated robot control tasks.

ABSTRACT

In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control. Deep Q-learning, however, still suffers from poor data-efficiency which is limiting with regard to real-world applications. We follow the idea of multi-step TD-learning to enhance data-efficiency while remaining off-policy by proposing two novel Temporal-Difference formulations: (1) Truncated Q-functions which represent the return for the first n steps of a policy rollout and (2) Shifted Q-functions, acting as the farsighted return after this truncated rollout. We prove that the combination of these short- and long-term predictions is a representation of the full return, leading to the Composite Q-learning algorithm. We show the efficacy of Composite Q-learning in the tabular case and compare our approach in the function-approximation setting with TD3, Model-based Value Expansion and TD3(Delta), which we introduce as an off-policy variant of TD(Delta). We show on three simulated robot tasks that Composite TD3 outperforms TD3 as well as state-of-the-art off-policy multi-step approaches in terms of data-efficiency.

Motivation & Objective

• Improve data-efficiency in off-policy deep reinforcement learning for real-world robot control applications.
• Address the limitations of deep Q-learning in sample efficiency despite recent advances in off-policy methods.
• Develop a novel off-policy multi-step TD-learning formulation that maintains stability while improving return estimation.
• Enable more accurate value function approximation by combining short-term and long-term return predictions.

Proposed method

• Introduce truncated Q-functions to represent the return over the first n steps of a policy rollout.
• Propose shifted Q-functions to model the farsighted return following the truncated rollout.
• Combine truncated and shifted Q-functions to form a composite representation of the full return.
• Derive a new off-policy learning algorithm, Composite Q-learning, based on this composite return formulation.
• Apply the method in both tabular and function-approximation settings, including a variant called Composite TD3.
• Introduce TD3(Delta) as an off-policy variant of TD(Delta) for comparative evaluation.

Experimental results

Research questions

• RQ1 Can combining short-term and long-term return predictions improve data efficiency in off-policy deep reinforcement learning?
• RQ2 How does the proposed Composite Q-learning method compare to TD3 and other off-policy multi-step methods in sample efficiency?
• RQ3 Does the use of truncated and shifted Q-functions lead to more stable and accurate value function learning in off-policy settings?
• RQ4 Can the composite return formulation maintain off-policy consistency while improving learning performance?

Key findings

• Composite Q-learning achieves superior data-efficiency compared to TD3 in tabular and function-approximation settings.
• Composite TD3 outperforms TD3 on three simulated robot control tasks in terms of learning efficiency and final performance.
• The method surpasses state-of-the-art off-policy multi-step approaches in data-efficiency, demonstrating improved sample utilization.
• The combination of truncated and shifted Q-functions provides a more accurate representation of the full return, enhancing learning stability and convergence.