[Paper Review] Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping

Lina Mezghani, Sainbayar Sukhbaatar+3 more|arXiv (Cornell University)|Jan 5, 2023

Intelligent Tutoring Systems and Adaptive Learning4 citations

TL;DR

This paper proposes a self-supervised reward shaping method for offline goal-conditioned reinforcement learning that learns dense rewards from pre-collected trajectories without any human-designed rewards. By training a reachability network, extracting representative states, and building a state-space graph, the method enables efficient learning of long-horizon policies, significantly outperforming prior hindsight relabeling baselines on continuous control tasks including UMaze, RoboYoga Walker, and Pusher.

ABSTRACT

Developing agents that can execute multiple skills by learning from pre-collected datasets is an important problem in robotics, where online interaction with the environment is extremely time-consuming. Moreover, manually designing reward functions for every single desired skill is prohibitive. Prior works targeted these challenges by learning goal-conditioned policies from offline datasets without manually specified rewards, through hindsight relabelling. These methods suffer from the issue of sparsity of rewards, and fail at long-horizon tasks. In this work, we propose a novel self-supervised learning phase on the pre-collected dataset to understand the structure and the dynamics of the model, and shape a dense reward function for learning policies offline. We evaluate our method on three continuous control tasks, and show that our model significantly outperforms existing approaches, especially on tasks that involve long-term planning.

Motivation & Objective

• To enable offline, unsupervised learning of multi-task goal-conditioned policies from pre-collected datasets without any reward engineering.
• To address the challenge of sparse rewards in long-horizon tasks that hinder policy learning in prior offline RL methods.
• To develop a self-supervised stage that learns environment dynamics and structure from trajectories to shape dense, meaningful rewards.
• To enable effective policy training on low-quality datasets, such as those collected with random policies, without expert demonstrations.
• To eliminate the need for additional environment interactions or manually designed reward functions at any stage of the pipeline.

Proposed method

• Train a reachability network to estimate local distances in the state space using transitions from the offline dataset.
• Extract a set of representative states that cover the state space to serve as landmarks for global distance approximation.
• Construct a graph over the representative states to model global distances in the state space using shortest-path computations.
• Use the graph to generate dense rewards during policy training based on the shortest-path distance to the goal.
• Augment the dataset with intermediate transitions along graph paths to improve policy generalization and training stability.
• Train a goal-conditioned policy using the self-supervised dense rewards and augmented transitions, without any reward function or environment interaction.

Experimental results

Research questions

• RQ1 Can self-supervised representation learning from offline trajectories enable dense reward shaping without human-designed rewards?
• RQ2 Does dense reward shaping improve learning efficiency and performance on long-horizon goal-conditioned control tasks?
• RQ3 Can the method train effective policies from low-quality datasets, such as those collected with random policies?
• RQ4 How does the proposed method compare to hindsight relabeling baselines like HER and Actionable Models in offline, goal-conditioned RL?
• RQ5 Can the method generalize to complex, continuous control environments without any reward supervision during data collection or training?

Key findings

• The proposed method significantly outperforms HER and Actionable Models on the UMaze task, especially for goals far from the initial state, demonstrating improved long-horizon planning.
• On the RoboYoga Walker task, the method achieves superior performance compared to baselines, while Actionable Models fails to improve over standard HER, indicating better sample efficiency and generalization.
• On the Pusher task, the method achieves lower average, hand, and puck distances to the goal than all baselines, showing precise and sequential control capabilities.
• The method learns effective policies even from datasets collected with a random policy, proving robustness to low-quality data and eliminating the need for expert demonstrations.
• The graph-based dense reward shaping enables faster convergence and more stable training compared to sparse reward methods, particularly in complex environments.
• The method is the first to enable fully unsupervised, offline goal-conditioned policy learning without any reward function at any stage—data collection, training, or evaluation.