[Paper Review] CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning

Ossama Ahmed, Frederik Träuble+6 more|arXiv (Cornell University)|Oct 8, 2020

Reinforcement Learning in Robotics42 references31 citations

TL;DR

CausalWorld introduces a parametrized robotic manipulation benchmark that enables interventions on environmental variables to study causal structure learning and transfer in RL, with curricula and sim-to-real transfer via the TriFinger platform.

ABSTRACT

Despite recent successes of reinforcement learning (RL), it remains a challenge for agents to transfer learned skills to related environments. To facilitate research addressing this problem, we propose CausalWorld, a benchmark for causal structure and transfer learning in a robotic manipulation environment. The environment is a simulation of an open-source robotic platform, hence offering the possibility of sim-to-real transfer. Tasks consist of constructing 3D shapes from a given set of blocks - inspired by how children learn to build complex structures. The key strength of CausalWorld is that it provides a combinatorial family of such tasks with common causal structure and underlying factors (including, e.g., robot and object masses, colors, sizes). The user (or the agent) may intervene on all causal variables, which allows for fine-grained control over how similar different tasks (or task distributions) are. One can thus easily define training and evaluation distributions of a desired difficulty level, targeting a specific form of generalization (e.g., only changes in appearance or object mass). Further, this common parametrization facilitates defining curricula by interpolating between an initial and a target task. While users may define their own task distributions, we present eight meaningful distributions as concrete benchmarks, ranging from simple to very challenging, all of which require long-horizon planning as well as precise low-level motor control. Finally, we provide baseline results for a subset of these tasks on distinct training curricula and corresponding evaluation protocols, verifying the feasibility of the tasks in this benchmark.

Motivation & Objective

• Motivate and enable research on out-of-distribution generalization in RL through a controllable causal environment.
• Provide a large, parameterizable suite of robotic manipulation tasks sharing causal structure.
• Allow interventions on environment parameters to study different generalization axes and curricula.
• Offer a unified success metric and evaluation protocols to compare learning algorithms across tasks.

Proposed method

• Define a parametrized family of tasks built from 3D shapes using blocks to form goal structures.
• Expose a wide set of causal variables (e.g., masses, colors, shapes, gravity) and allow do-interventions on them.
• Support multiple observation modes (structured low-dimensional and pixel-based) and various action spaces for the TriFinger robot.
• Introduce training and evaluation spaces (ATS and ES) to enable curricula and out-of-distribution evaluation.
• Provide task generators (e.g., Pushing, Picking, Pick and Place, Stacking2, Towers, etc.) for diverse goals.
• Benchmark baseline model-free RL methods (PPO, SAC, TD3) under different curricula and evaluation protocols.

Experimental results

Research questions

• RQ1 How does altering environmental causal variables during training affect transfer to unseen tasks?
• RQ2 Can a unified success metric and curriculum-driven interventions disentangle in-distribution and out-of-distribution generalization in robotic manipulation?
• RQ3 What are the limitations of current model-free RL methods on complex, multi-object goal shapes under various curricula?
• RQ4 How do sim-to-real considerations impact learning when transferring policies to the real TriFinger platform?

Key findings

• Model-free RL methods solve simple single-block tasks under sufficient training but struggle with multi-block stacking tasks.
• Curricula that randomize goal shapes or environmental parameters substantially affect generalization performance, with extreme randomization hindering learning.
• Some generalization to new initial poses occurs under goal-shape randomization, but extreme domain randomization can impede learning.
• A unified, parameterized benchmark like CausalWorld enables explicit evaluation of in-distribution vs. out-of-distribution generalization across axes (e.g., mass, friction, color).
• Baseline results verify feasibility of the tasks and highlight the need for inductive biases or structured methods for complex, multi-object manipulation.