[Paper Review] Making Deep Q-learning methods robust to time discretization

Corentin Tallec, Léonard Blier+1 more|arXiv (Cornell University)|Jan 28, 2019

Reinforcement Learning in Robotics17 references33 citations

TL;DR

The paper shows that Q-learning approaches collapse with small time steps and proposes a continuous-time robust off-policy algorithm called Deep Advantage Updating (DAU) that remains effective across a range of time discretizations, with theoretical support and empirical validation.

ABSTRACT

Despite remarkable successes, Deep Reinforcement Learning (DRL) is not robust to hyperparameterization, implementation details, or small environment changes (Henderson et al. 2017, Zhang et al. 2018). Overcoming such sensitivity is key to making DRL applicable to real world problems. In this paper, we identify sensitivity to time discretization in near continuous-time environments as a critical factor; this covers, e.g., changing the number of frames per second, or the action frequency of the controller. Empirically, we find that Q-learning-based approaches such as Deep Q- learning (Mnih et al., 2015) and Deep Deterministic Policy Gradient (Lillicrap et al., 2015) collapse with small time steps. Formally, we prove that Q-learning does not exist in continuous time. We detail a principled way to build an off-policy RL algorithm that yields similar performances over a wide range of time discretizations, and confirm this robustness empirically.

Motivation & Objective

• Identify sensitivity of DRL to time discretization in near-continuous environments.
• Prove the collapse of Q-learning to the V-function as time discretization goes to zero.
• Develop and justify an off-policy algorithm that remains robust to varying time steps.
• Provide theoretical foundations for continuous-time limits and practical algorithms (DAU).
• Empirically validate robustness of the proposed method across diverse environments.

Proposed method

• Frame near-continuous environments as discretizations of continuous-time MDPs with delta t, and scale rewards and discounts to preserve meaningful limits.
• Introduce Advantage Updating by defining A_delta_t(s,a) = (Q_delta_t(s,a) - V_delta_t(s)) / delta t and enforce consistency with V to obtain a learnable, invariant ranking signal.
• Parameterize Q as Q_theta(s,a) = V_theta(s) + delta t * A_psi(s,a) and enforce A(s, pi(s)) = 0 for identifiability.
• Use a reparameterization A_psi(s,a) = barA_psi(s,a) - barA_psi(s, pi(s)) to ensure consistent action rankings.
• Adopt timestep-invariant exploration via a continuous-time limit of Ornstein-Uhlenbeck noise for continuous actions and a discretized approach for discrete actions.
• Derive learning-rate scalings eta_V_delta_t, eta_A_delta_t proportional to delta t to guarantee a well-defined continuous-time limit.

Experimental results

Research questions

• RQ1 Can Q-learning-based methods be made invariant to time discretization in near-continuous environments?
• RQ2 What theoretical limits exist for Q-function-based methods as delta t tends to zero?
• RQ3 Can an off-policy algorithm be designed to preserve performance across a wide range of time steps?
• RQ4 How should exploration and learning-rate schedules be scaled to achieve a continuous-time limit?
• RQ5 Do the proposed methods empirically exhibit robustness across standard control benchmarks when delta t varies?

Key findings

• Q-learning collapses to the V-function in near-continuous time, making it ill-behaved in continuous time.
• A delta t-scaled advantage (A_delta_t) retains action-ranking information and has a non-degenerate limit as delta t -> 0.
• A reparameterized Q via V and delta t A ensures identifiability and stable learning across time steps.
• A timestep-invariant exploration scheme leads to meaningful continuous-time limits for both discrete and continuous actions.
• Deep Advantage Updating (DAU) shows improved robustness to time discretization across several control tasks compared to vanilla DQN or DDPG.