[Paper Review] Memory Based Trajectory-conditioned Policies for Learning from Sparse Rewards

Yijie Guo, Jongwook Choi+5 more|arXiv (Cornell University)|Jul 24, 2019

Reinforcement Learning in Robotics14 citations

TL;DR

This paper proposes DTSIL, a memory-based trajectory-conditioned policy that improves exploration in sparse-reward reinforcement learning by imitating diverse past trajectories from a memory buffer. By conditioning policy learning on multiple demonstrated trajectories and using sequence-to-sequence modeling with attention, DTSIL enables flexible, exploratory behavior that outperforms count-based exploration and self-imitation learning on hard Atari environments like Montezuma’s Revenge and Pitfall, achieving state-of-the-art performance without expert demonstrations or arbitrary resets.

ABSTRACT

Reinforcement learning with sparse rewards is challenging because an agent can rarely obtain non-zero rewards and hence, gradient-based optimization of parameterized policies can be incremental and slow. Recent work demonstrated that using a memory buffer of previous successful trajectories can result in more effective policies. However, existing methods may overly exploit past successful experiences, which can encourage the agent to adopt sub-optimal and myopic behaviors. In this work, instead of focusing on good experiences with limited diversity, we propose to learn a trajectory-conditioned policy to follow and expand diverse past trajectories from a memory buffer. Our method allows the agent to reach diverse regions in the state space and improve upon the past trajectories to reach new states. We empirically show that our approach significantly outperforms count-based exploration methods (parametric approach) and self-imitation learning (parametric approach with non-parametric memory) on various complex tasks with local optima. In particular, without using expert demonstrations or resetting to arbitrary states, we achieve the state-of-the-art scores under five billion number of frames, on challenging Atari games such as Montezuma's Revenge and Pitfall.

Motivation & Objective

• To address the challenge of sparse-reward reinforcement learning where gradient updates are slow and agents often get stuck in local optima.
• To overcome the limitations of existing methods that overly exploit a few successful trajectories, leading to myopic and suboptimal behaviors.
• To improve sample efficiency and long-horizon exploration by leveraging diverse past experiences stored in a memory buffer.
• To enable agents to discover novel states and reach higher rewards by generating new trajectories through editing and augmenting stored demonstrations.
• To achieve state-of-the-art performance on challenging Atari games without using expert demonstrations or arbitrary state resets.

Proposed method

• The method uses a trajectory-conditioned policy where a sequence of states from past successful trajectories serves as input context for action generation.
• A sequence-to-sequence model with an attention mechanism learns to translate demonstration trajectories into new action sequences, enabling stochastic and flexible policy behavior.
• The policy is trained using a supervised learning objective on diverse trajectories stored in a memory buffer, promoting exploration beyond the most frequently visited paths.
• The memory buffer stores full trajectories from past episodes, and the policy is conditioned on any such trajectory to generate new, exploratory behaviors.
• The approach combines count-based exploration over trajectory-level demonstrations with imitation learning to guide the agent toward rarely visited but potentially high-reward states.
• A hierarchical variant is explored where a high-level policy proposes sub-goals based on buffer trajectories, and a low-level policy executes actions to reach them, improving generalization to unseen environments.

Experimental results

Research questions

• RQ1 Can imitating diverse past trajectories from a memory buffer improve exploration in sparse-reward environments compared to standard count-based or self-imitation methods?
• RQ2 Does conditioning a policy on multiple diverse trajectories reduce myopic behavior and help avoid premature convergence to suboptimal policies?
• RQ3 Can a sequence-to-sequence model with attention effectively generate new, high-reward trajectories by editing stored demonstrations?
• RQ4 To what extent can the method generalize to unseen environments without expert demonstrations or reset-to-state mechanisms?
• RQ5 How does the method perform on long-horizon tasks with local optima, such as Montezuma’s Revenge and Pitfall?

Key findings

• DTSIL achieves state-of-the-art performance on Montezuma’s Revenge and Pitfall at five billion frames, outperforming prior methods without using expert demonstrations or arbitrary resets.
• The method significantly improves sample efficiency and exploration by leveraging diverse trajectories, enabling the agent to reach the left-most room in Pitfall and collect many diamonds, which prior SOTA failed to do.
• In the random maze environment, the hierarchical DTSIL variant generalizes to unseen mazes, achieving higher average episode rewards on the test set compared to PPO+EXP.
• DTSIL outperforms both count-based exploration and self-imitation learning across multiple tasks with local optima and stochastic dynamics.
• The use of diverse trajectory conditioning leads to better long-term return and avoids premature convergence to suboptimal policies.
• Empirical results show that the method is robust and effective in environments requiring reasoning over long horizons and sparse rewards.