[Paper Review] Solving Rubik's Cube with a Robot Hand

OpenAI, Ilge Akkaya+17 more|arXiv (Cornell University)|Oct 16, 2019

Domain Adaptation and Few-Shot Learning111 references630 citations

TL;DR

The paper trains control policies and vision state estimators entirely in simulation using automatic domain randomization (ADR) to solve a Rubik’s cube with a humanoid robot hand, demonstrating effective sim2real transfer.

ABSTRACT

We demonstrate that models trained only in simulation can be used to solve a manipulation problem of unprecedented complexity on a real robot. This is made possible by two key components: a novel algorithm, which we call automatic domain randomization (ADR) and a robot platform built for machine learning. ADR automatically generates a distribution over randomized environments of ever-increasing difficulty. Control policies and vision state estimators trained with ADR exhibit vastly improved sim2real transfer. For control policies, memory-augmented models trained on an ADR-generated distribution of environments show clear signs of emergent meta-learning at test time. The combination of ADR with our custom robot platform allows us to solve a Rubik's cube with a humanoid robot hand, which involves both control and state estimation problems. Videos summarizing our results are available: https://openai.com/blog/solving-rubiks-cube/

Motivation & Objective

• Demonstrate solving a Rubik’s Cube with a five-finger humanoid hand using only simulated data for training.
• Introduce automatic domain randomization (ADR) to generate growing, diverse training environments for both control policies and vision models.
• Investigate why ADR-trained policies transfer to real hardware and whether emergent meta-learning occurs.
• Build a robot platform and an accompanying simulation pipeline that closely model the manipulation and sensing tasks involved.

Proposed method

• Develop a novel ADR algorithm that gradually expands a distribution over randomized simulation environments.
• Train a memory-augmented control policy (LSTM-based) via reinforcement learning to solve randomized tasks.
• Train a vision-based Rubik’s cube pose estimator using rendered images from the ADR-generated simulations.
• Model a detailed MuJoCo-based simulation of the Shadow Dexterous Hand and a 3D Rubik’s cube to bridge sim-to-real gaps.
• Use a distributed, ADR-driven training pipeline with centralized storage (Redis) for parameters, data, and performance buffers.

Experimental results

Research questions

• RQ1 Can models trained exclusively in simulation with ADR transfer effectively to real-world manipulation of a Rubik’s cube by a humanoid hand?
• RQ2 Does memory-augmented policy training under ADR exhibit emergent meta-learning during real-world deployment?
• RQ3 How does automatic domain randomization compare to manual domain randomization in achieving robust sim2real transfer?
• RQ4 What are the key design considerations in the physical and simulation platforms to support ADR for complex manipulation tasks?

Key findings

• ADR enables successful sim2real transfer for a complex manipulation task involving a Rubik’s cube and a five-finger hand.
• Memory-augmented policies trained with an expanding ADR distribution show signs of emergent meta-learning at test time.
• Vision state estimators trained under ADR can predict cube pose and face angles from RGB camera feeds in the real world.
• Systematic augmentation of simulation realism (hand dynamics, cube model, sensor noise) improves transfer performance.
• A dedicated robot platform and a scalable, distributed ADR training pipeline support efficient training and evaluation.