[Paper Review] Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation

Peide Huang, Mengdi Xu+4 more|arXiv (Cornell University)|Oct 18, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes GRADIENT, a curriculum reinforcement learning framework that formulates task distribution interpolation as an optimal transport problem using Wasserstein barycenters, enabling smooth, gradual policy transfer from source to target tasks. By leveraging a task-dependent contextual distance metric and handling both discrete and continuous contexts, GRADIENT achieves superior learning efficiency and asymptotic performance compared to SOTA baselines in locomotion and manipulation tasks.

ABSTRACT

Curriculum Reinforcement Learning (CRL) aims to create a sequence of tasks, starting from easy ones and gradually learning towards difficult tasks. In this work, we focus on the idea of framing CRL as interpolations between a source (auxiliary) and a target task distribution. Although existing studies have shown the great potential of this idea, it remains unclear how to formally quantify and generate the movement between task distributions. Inspired by the insights from gradual domain adaptation in semi-supervised learning, we create a natural curriculum by breaking down the potentially large task distributional shift in CRL into smaller shifts. We propose GRADIENT, which formulates CRL as an optimal transport problem with a tailored distance metric between tasks. Specifically, we generate a sequence of task distributions as a geodesic interpolation (i.e., Wasserstein barycenter) between the source and target distributions. Different from many existing methods, our algorithm considers a task-dependent contextual distance metric and is capable of handling nonparametric distributions in both continuous and discrete context settings. In addition, we theoretically show that GRADIENT enables smooth transfer between subsequent stages in the curriculum under certain conditions. We conduct extensive experiments in locomotion and manipulation tasks and show that our proposed GRADIENT achieves higher performance than baselines in terms of learning efficiency and asymptotic performance.

Motivation & Objective

• To address the challenge of large distributional shifts in curriculum reinforcement learning (CRL) by breaking them into smaller, manageable transitions.
• To formalize curriculum generation as a geodesic interpolation between source and target task distributions using optimal transport theory.
• To develop a method that handles nonparametric, continuous, and discrete context spaces without requiring parametric distribution assumptions.
• To ensure smooth policy transfer between curriculum stages through theoretical guarantees on transfer performance.
• To improve learning efficiency and asymptotic performance in complex RL tasks such as navigation and manipulation.

Proposed method

• Formulates CRL as an optimal transport problem using the Wasserstein barycenter to generate intermediate task distributions between source and target.
• Introduces a task-dependent contextual distance metric, $ d^\pi(c_i, c_j) = |J^\pi(c_i) - J^\pi(c_j)| $, based on estimated return differences to reflect task similarity.
• Employs a learned embedding space via autoencoders to map contexts into a latent space where Euclidean distances approximate Wasserstein distances, enabling interpolation in free-support settings.
• Uses a two-stage algorithm: first, estimate the policy value function $ J^\pi(c) $ via Gaussian process regression from collected trajectories; second, train encoder-decoder networks to embed the contextual distance metric.
• Applies gradual interpolation via increasing $ \alpha \in [0,1] $, generating a sequence of task distributions as barycenters between source and target.
• Employs a curriculum loop where the agent is trained on each interpolated distribution before updating the policy and refining the distance metric.

Experimental results

Research questions

• RQ1 Can optimal transport be effectively used to generate a natural, gradual curriculum in CRL by interpolating between source and target task distributions?
• RQ2 How can a task-dependent, contextually meaningful distance metric be defined to reflect the difficulty gap between tasks in both continuous and discrete settings?
• RQ3 Can the proposed method ensure smooth policy transfer between consecutive curriculum stages, and under what conditions is this guaranteed?
• RQ4 Does the use of Wasserstein barycenters with learned embeddings outperform linear interpolation and existing CRL baselines in terms of learning efficiency and final performance?
• RQ5 How does the method handle nonparametric and implicit task distributions without assuming parametric forms?

Key findings

• GRADIENT achieves higher learning efficiency than SOTA CRL baselines, with faster convergence in locomotion and manipulation tasks.
• The method demonstrates superior asymptotic performance, achieving higher final return in complex environments such as the U-shaped maze.
• In the U-shaped maze with obstacles, GRADIENT's interpolation avoids the pitfalls of linear interpolation by covering intermediate, non-zero-probability start positions.
• Theoretical analysis shows that under certain conditions, GRADIENT ensures smooth policy transfer between curriculum stages, minimizing performance drops.
• The use of learned distance embeddings enables effective interpolation even in free-support, continuous context spaces where $ l_2 $ distance fails to capture structural task differences.
• Empirical results show that as the policy improves over stages, the quality of the interpolated task distributions increases, leading to better overall curriculum progression.