[Paper Review] Disentangling Dynamics and Returns: Value Function Decomposition with Future Prediction

Hongyao Tang, Jianye Hao+5 more|arXiv (Cornell University)|May 27, 2019

Financial Markets and Investment StrategiesEconomics, Econometrics and Finance32 references3 citations

TL;DR

This paper proposes Value Function Decomposition with Future Prediction (VDFP), a deep reinforcement learning algorithm that decomposes value functions into separate dynamics and return components via future trajectory prediction. By using a conditional VAE for dynamics modeling and a convex return head, VDFP achieves state-of-the-art performance in MuJoCo continuous control tasks, especially under delayed reward settings, outperforming DDPG by 2x–4x in final returns.

ABSTRACT

Value functions are crucial for model-free Reinforcement Learning (RL) to obtain a policy implicitly or guide the policy updates. Value estimation heavily depends on the stochasticity of environmental dynamics and the quality of reward signals. In this paper, we propose a two-step understanding of value estimation from the perspective of future prediction, through decomposing the value function into a reward-independent future dynamics part and a policy-independent trajectory return part. We then derive a practical deep RL algorithm from the above decomposition, consisting of a convolutional trajectory representation model, a conditional variational dynamics model to predict the expected representation of future trajectory and a convex trajectory return model that maps a trajectory representation to its return. Our algorithm is evaluated in MuJoCo continuous control tasks and shows superior results under both common settings and delayed reward settings.

Motivation & Objective

• To address the entanglement of environmental dynamics and reward signals in value function estimation, which degrades performance in stochastic and delayed-reward environments.
• To develop a model-free RL algorithm that decouples value estimation into two distinct, learnable components: future dynamics prediction and trajectory return estimation.
• To improve sample efficiency and robustness in challenging settings such as delayed rewards by training dynamics and return models independently.
• To provide a new theoretical and practical framework for value function estimation based on future prediction, inspired by human cognitive processes.
• To validate the method empirically on standard MuJoCo benchmarks and ablate each component for reproducibility and insight.

Proposed method

• The value function is re-expressed as a composition of a reward-independent predictive dynamics function and a policy-independent trajectory return function.
• A convolutional neural network encodes trajectories into compact representations for downstream modeling.
• A conditional variational autoencoder (VAE) models the distribution of future trajectory representations, with clipped generation noise used to approximate the expected future state-action trajectory.
• The dynamics model is trained to predict the expected future representation, decoupled from reward signals.
• A convex trajectory return model maps trajectory representations to their discounted cumulative returns, trained independently from the dynamics model.
• The overall algorithm uses off-policy training with experience replay, leveraging the decoupled components for stable and efficient learning.

Experimental results

Research questions

• RQ1 Can value function estimation be effectively decomposed into separate dynamics and return components through future prediction?
• RQ2 Does decoupling dynamics and return modeling improve learning stability and performance in environments with stochastic dynamics or delayed rewards?
• RQ3 Can a conditional VAE with clipped noise generation effectively model expected future trajectories while remaining robust to reward signal irregularities?
• RQ4 How does the proposed method compare to existing SOTA algorithms in standard and delayed-reward MuJoCo environments?
• RQ5 What is the contribution of each component (dynamics, return, representation) to the overall performance, and how robust is the method under distributional shift?

Key findings

• VDFP achieves a final return of 5818.60 ± 336.25 on HalfCheetah-v1 under standard training, outperforming DDPG and other baselines.
• Under delayed reward settings with a 16-step delay, VDFP achieves 5712.55 ± 233.74 return on HalfCheetah-v1, outperforming DDPG by over 2x in final performance.
• With a 128-step delay, VDFP maintains a return of 4752.84 ± 328.75 on HalfCheetah-v1, demonstrating strong robustness compared to DDSR, which fails to learn effectively.
• Ablation studies show that removing the conditional VAE or the convex return model significantly degrades performance, confirming the importance of both components.
• The method shows consistent superiority in both learning speed and final performance across all tested environments, including Walker2d-v1 and Hopper-v1.
• The use of off-policy training does not degrade performance, suggesting that the deterministic policy and decoupled architecture reduce on-policy constraints.