[Paper Review] Estimating Q(s,s') with Deep Deterministic Dynamics Gradients

Ashley D. Edwards, Himanshu Sahni+8 more|arXiv (Cornell University)|Feb 21, 2020

Reinforcement Learning in Robotics40 references4 citations

TL;DR

This paper introduces Q(s,s'), a novel value function that evaluates the utility of transitioning from state s to a neighboring state s' and acting optimally thereafter, decoupling actions from values. By training a forward dynamics model via Deep Deterministic Dynamics Gradients (D3G) to predict next states that maximize Q(s,s'), the method enables off-policy learning from purely observational data, even from random or sub-optimal policies, with improved transferability and efficiency in redundant action spaces.

ABSTRACT

In this paper, we introduce a novel form of value function, $Q(s, s')$, that expresses the utility of transitioning from a state $s$ to a neighboring state $s'$ and then acting optimally thereafter. In order to derive an optimal policy, we develop a forward dynamics model that learns to make next-state predictions that maximize this value. This formulation decouples actions from values while still learning off-policy. We highlight the benefits of this approach in terms of value function transfer, learning within redundant action spaces, and learning off-policy from state observations generated by sub-optimal or completely random policies. Code and videos are available at http://sites.google.com/view/qss-paper.

Motivation & Objective

• To develop a value function that expresses the utility of state transitions independent of actions, enabling decoupled value and policy learning.
• To enable off-policy reinforcement learning using only state observations, without access to demonstrator actions.
• To improve transferability and sample efficiency in environments with redundant or high-dimensional action spaces.
• To train a forward dynamics model that predicts next states maximizing future return, without explicit action supervision.

Proposed method

• Proposes Q(s,s') as a value function representing the return of transitioning from s to s' and acting optimally thereafter, defined by Q(s,s') = r + γ max_{s''} Q(s',s'').
• Introduces Deep Deterministic Dynamics Gradients (D3G), a method to train a forward dynamics model τ(s) → s' that maximizes Q(s,τ(s)) without conditioning on actions.
• Uses a differentiable dynamics model to perform gradient updates on next-state predictions based on value function targets, enabling end-to-end off-policy learning.
• Employs a two-stage learning process: first, Q(s,s') is learned via bootstrapping; second, τ(s) is trained to predict next states that maximize Q(s,τ(s)).
• Leverages the fact that multiple actions may lead to the same state transition, naturally folding redundant actions into a single value estimate.
• Applies the method in both tabular and continuous control environments, including MuJoCo tasks, using function approximation and experience replay.

Experimental results

Research questions

• RQ1 Can a value function that depends only on state transitions (Q(s,s')) outperform traditional Q(s,a) in terms of sample efficiency and transferability?
• RQ2 Can a forward dynamics model be trained to generate next-state predictions that maximize future return without action supervision?
• RQ3 Can Q(s,s') and D3G learn effective policies from purely observational data, even when demonstrations are random or sub-optimal?
• RQ4 Does decoupling value from actions improve transfer across environments with different action representations but identical dynamics?

Key findings

• Q(s,s') achieves performance comparable to Q(s,a) in standard tabular environments, demonstrating the viability of action-agnostic value learning.
• The method successfully learns policies from purely observational data, including from completely random demonstrations, without access to expert actions.
• In environments with redundant actions, Q(s,s') naturally collapses equivalent transitions into a single value estimate, improving sample efficiency.
• D3G enables off-policy training of dynamics models to maximize return, achieving high sample efficiency in MuJoCo control tasks.
• The approach enables effective value transfer across tasks with identical dynamics but different action spaces, requiring only retraining of the inverse dynamics model upon transfer.
• The method supports automatic goal generation and planning from observation-only data, outperforming imitation-from-observation baselines that require action labels.