[Paper Review] Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

Josh Tobin, Rachel Fong+4 more|arXiv (Cornell University)|Mar 20, 2017

Domain Adaptation and Few-Shot LearningComputer Science46 references206 citations

TL;DR

The paper demonstrates that object detectors trained entirely in simulation with domain randomization can transfer to the real world and enable grasping in clutter without real-world training data.

ABSTRACT

Bridging the 'reality gap' that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator. With enough variability in the simulator, the real world may appear to the model as just another variation. We focus on the task of object localization, which is a stepping stone to general robotic manipulation skills. We find that it is possible to train a real-world object detector that is accurate to $1.5$cm and robust to distractors and partial occlusions using only data from a simulator with non-realistic random textures. To demonstrate the capabilities of our detectors, we show they can be used to perform grasping in a cluttered environment. To our knowledge, this is the first successful transfer of a deep neural network trained only on simulated RGB images (without pre-training on real images) to the real world for the purpose of robotic control.

Motivation & Objective

• Motivate faster, cheaper robotic data collection by training in simulation to bridge the reality gap.
• Propose domain randomization as a method to generalize to real-world visuals by exposing the model to highly varied synthetic textures and scene configurations.
• Show that a detector trained solely on randomized simulated RGB images can localize objects with high precision in the real world.
• Demonstrate that the learned detectors can be used to perform grasping in cluttered real-world environments.

Proposed method

• Train a CNN-based object detector (modified VGG-16) to map a single monocular frame to 3D object coordinates.
• Randomize numerous scene aspects during simulation, including object counts, textures, lighting, camera pose, and noise, using non-realistic textures.
• Render with MuJoCo’s renderer to generate hundreds of thousands of labeled samples.
• Use pretraining weights (ImageNet) or random initialization and optimize with Adam on L2 loss for object center coordinates.
• Evaluate localization accuracy on real webcam images under distractors and occlusion, and perform robotic grasping experiments with a Fetch robot.

Experimental results

Research questions

• RQ1 Can domain randomization enable sim-to-real transfer for precise object localization from a single RGB image without real-world labeled data?
• RQ2 What elements of domain randomization (textures, camera pose, lighting, distractors) are critical for robust real-world performance?
• RQ3 Is pretraining on real images essential for transfer, or can random initialization with sufficient synthetic data suffice?
• RQ4 Can detectors trained in simulation support real robotic grasping in cluttered scenes?

Key findings

• Detectors trained only on simulated data with domain randomization localize objects in the real world to around 1.5 cm accuracy on average.
• Performance remains robust to distractors and partial occlusions in real scenes.
• Pretraining on real images is not strictly necessary with enough synthetic data; random initialization can match pre-trained performance, though pretraining helps in low-data regimes.
• In robotics experiments, the detectors enabled successful grasping in clutter on a Fetch robot in 38 of 40 trials.
• The approach can generalize to real-world objects with non-uniform textures (e.g., a can from the YCB dataset) with successful picks in most trials.