[Paper Review] C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks

Tianjun Zhang, Benjamin Eysenbach+3 more|arXiv (Cornell University)|Oct 22, 2021

Reinforcement Learning in Robotics38 references4 citations

TL;DR

C-Planning proposes an automatic curriculum for goal-reaching tasks by using training-time graph search to generate intermediate waypoints, improving sample efficiency and enabling learning of long-horizon navigation and manipulation tasks without test-time planning. It achieves state-of-the-art performance by integrating search into policy training via expectation maximization, outperforming prior methods that rely on test-time search or manual curriculum design.

ABSTRACT

Goal-conditioned reinforcement learning (RL) can solve tasks in a wide range of domains, including navigation and manipulation, but learning to reach distant goals remains a central challenge to the field. Learning to reach such goals is particularly hard without any offline data, expert demonstrations, and reward shaping. In this paper, we propose an algorithm to solve the distant goal-reaching task by using search at training time to automatically generate a curriculum of intermediate states. Our algorithm, Classifier-Planning (C-Planning), frames the learning of the goal-conditioned policies as expectation maximization: the E-step corresponds to planning an optimal sequence of waypoints using graph search, while the M-step aims to learn a goal-conditioned policy to reach those waypoints. Unlike prior methods that combine goal-conditioned RL with graph search, ours performs search only during training and not testing, significantly decreasing the compute costs of deploying the learned policy. Empirically, we demonstrate that our method is more sample efficient than prior methods. Moreover, it is able to solve very long horizons manipulation and navigation tasks, tasks that prior goal-conditioned methods and methods based on graph search fail to solve.

Motivation & Objective

• To address the challenge of learning to reach distant goals in goal-conditioned reinforcement learning without expert demonstrations or reward shaping.
• To improve sample efficiency in long-horizon control tasks such as multi-object manipulation and complex navigation.
• To eliminate the need for test-time planning by embedding search into the training process.
• To automatically generate a curriculum of intermediate waypoints that improves policy data quality.
• To enable deployment of learned policies with minimal inference latency by avoiding online planning.

Proposed method

• C-Planning frames goal-conditioned RL as an expectation maximization (EM) process, where the E-step uses graph search to plan optimal waypoint sequences for reaching distant goals.
• The M-step trains a goal-conditioned policy to reach these waypoints, improving data quality and policy generalization.
• Waypoints are sampled from a distribution that converges to the states visited by the optimal policy, as predicted by theory.
• The method uses variational inference to derive a lower bound on goal-reaching probability, enabling automatic curriculum generation.
• It integrates seamlessly with existing goal-conditioned RL algorithms, requiring no architectural changes.
• Test-time inference directly uses the final goal without intermediate waypoints, eliminating planning overhead.

Experimental results

Research questions

• RQ1 Can training-time search be used to automatically generate a curriculum of intermediate goals that improves sample efficiency in long-horizon tasks?
• RQ2 Does embedding search into the training process outperform test-time planning methods in terms of sample efficiency and deployment latency?
• RQ3 Can an automatic curriculum emerge from variational inference without human-designed distance functions or expert demonstrations?
• RQ4 How does the distribution of waypoints evolve during training, and does it align with the optimal policy's state visitation?
• RQ5 Can C-Planning solve complex, high-dimensional manipulation tasks that prior goal-conditioned RL methods fail to learn?

Key findings

• C-Planning achieves state-of-the-art performance on 2D navigation and 18D robotic manipulation tasks, solving tasks that prior methods fail to learn.
• The method improves sample efficiency compared to baseline goal-conditioned RL algorithms, as shown by faster convergence in training curves.
• C-Planning is 1.74× faster than SoRB with four waypoints and 4.71× faster with 16 waypoints in the Push-Wall environment, demonstrating low deployment latency.
• Gradient analysis shows C-Planning increases the critic gradient norm and reduces the variance of the actor gradient, suggesting better learning signals.
• Waypoint sampling evolves from uniform distribution to align with the optimal policy’s state trajectory, validating theoretical predictions.
• The method is robust to hyperparameter choices, such as the number of waypoints, maintaining strong performance across different settings.