[Paper Review] ACuTE: Automatic Curriculum Transfer from Simple to Complex Environments

Yash Shukla, Christopher Thierauf+3 more|arXiv (Cornell University)|Apr 11, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

ACuTE proposes a novel framework for automatic curriculum transfer from low-fidelity (LF) to high-fidelity (HF) environments in reinforcement learning, where a curriculum optimized in a simple environment is mapped to a complex one without direct HF interaction. The method achieves faster convergence and improved jumpstart performance on challenging HF tasks, including Sim2Real transfer to a physical robot, outperforming baselines like domain adaptation and self-play.

ABSTRACT

Despite recent advances in Reinforcement Learning (RL), many problems, especially real-world tasks, remain prohibitively expensive to learn. To address this issue, several lines of research have explored how tasks, or data samples themselves, can be sequenced into a curriculum to learn a problem that may otherwise be too difficult to learn from scratch. However, generating and optimizing a curriculum in a realistic scenario still requires extensive interactions with the environment. To address this challenge, we formulate the curriculum transfer problem, in which the schema of a curriculum optimized in a simpler, easy-to-solve environment (e.g., a grid world) is transferred to a complex, realistic scenario (e.g., a physics-based robotics simulation or the real world). We present "ACuTE", Automatic Curriculum Transfer from Simple to Complex Environments, a novel framework to solve this problem, and evaluate our proposed method by comparing it to other baseline approaches (e.g., domain adaptation) designed to speed up learning. We observe that our approach produces improved jumpstart and time-to-threshold performance even when adding task elements that further increase the difficulty of the realistic scenario. Finally, we demonstrate that our approach is independent of the learning algorithm used for curriculum generation, and is Sim2Real transferable to a real world scenario using a physical robot.

Motivation & Objective

• To address the high computational cost of curriculum generation in complex, high-fidelity (HF) environments.
• To enable transfer of a curriculum schema from a low-fidelity (LF) environment to a HF environment without requiring direct interaction in the HF domain.
• To improve learning efficiency and convergence speed on difficult HF tasks, including those with added complexity.
• To demonstrate Sim2Real transferability using a physical TurtleBot robot.
• To show independence from the RL algorithm used to generate the curriculum.

Proposed method

• Generate and optimize a curriculum in a simplified, low-fidelity (LF) environment with reduced dynamics complexity.
• Map task parameters from the LF curriculum to corresponding tasks in the high-fidelity (HF) environment using a heuristic mapping function.
• Transfer only the curriculum schema—task parameters and sequence—while preserving inter-task relationships based on high-level task descriptors.
• Learn source tasks iteratively in the HF environment before training on the final target task.
• Use the same policy from the HF simulation for direct Sim2Real deployment on a physical robot without further learning.
• Ensure robustness by validating success only when physical actions (e.g., fiducial reading) confirm task execution.

Experimental results

Research questions

• RQ1 Can a curriculum optimized in a low-fidelity environment be effectively transferred to a high-fidelity environment to accelerate learning?
• RQ2 Does curriculum transfer improve learning speed and jumpstart performance on complex HF tasks, even when dynamics differ significantly?
• RQ3 Is the proposed method independent of the reinforcement learning algorithm used to generate the curriculum?
• RQ4 Can the policy learned via curriculum transfer be successfully deployed on a physical robot without further training?
• RQ5 How does curriculum transfer compare to domain adaptation and other baseline methods in terms of computational cost and learning efficiency?

Key findings

• ACuTE achieved faster convergence and improved time-to-threshold performance on the target HF task compared to domain adaptation, self-play, and teacher-student curriculum baselines.
• The method demonstrated superior jumpstart performance, even when additional complexity was introduced to the HF environment.
• The framework is independent of the RL algorithm used for curriculum generation, as shown by improved performance when using a different algorithm in the HF environment.
• Sim2Real transfer succeeded on a physical TurtleBot, with the policy completing object breaking, retrieval, and crafting tasks after direct deployment.
• The computational cost of ACuTE was significantly lower than baseline methods, with a sunk cost dominated by LF curriculum optimization and HF source task learning.
• Even with imperfect mapping between LF and HF environments, positive transfer was observed, indicating robustness to mapping inaccuracies.