[Paper Review] Scaling Robot Learning with Semantically Imagined Experience

Tianhe Yu, Ted Xiao+11 more|arXiv (Cornell University)|Feb 22, 2023

Topic Modeling4 citations

TL;DR

This paper proposes ROSIE, a method that uses text-to-image diffusion models to generate semantically meaningful, photorealistic data augmentations for robot learning without collecting real-world data. By inpainting objects, backgrounds, and distractors using natural language prompts, ROSIE enables policies to generalize to unseen tasks and robustly handle novel, cluttered environments, significantly improving success detection F1 scores on out-of-distribution scenarios from 0.19 to 0.57.

ABSTRACT

Recent advances in robot learning have shown promise in enabling robots to perform a variety of manipulation tasks and generalize to novel scenarios. One of the key contributing factors to this progress is the scale of robot data used to train the models. To obtain large-scale datasets, prior approaches have relied on either demonstrations requiring high human involvement or engineering-heavy autonomous data collection schemes, both of which are challenging to scale. To mitigate this issue, we propose an alternative route and leverage text-to-image foundation models widely used in computer vision and natural language processing to obtain meaningful data for robot learning without requiring additional robot data. We term our method Robot Learning with Semantically Imagened Experience (ROSIE). Specifically, we make use of the state of the art text-to-image diffusion models and perform aggressive data augmentation on top of our existing robotic manipulation datasets via inpainting various unseen objects for manipulation, backgrounds, and distractors with text guidance. Through extensive real-world experiments, we show that manipulation policies trained on data augmented this way are able to solve completely unseen tasks with new objects and can behave more robustly w.r.t. novel distractors. In addition, we find that we can improve the robustness and generalization of high-level robot learning tasks such as success detection through training with the diffusion-based data augmentation. The project's website and videos can be found at diffusion-rosie.github.io

Motivation & Objective

• To address the challenge of scaling real-world robot data collection, which is costly and time-intensive due to reliance on human demonstrations or engineered autonomous systems.
• To explore whether off-the-shelf text-to-image diffusion models can generate diverse, realistic, and semantically meaningful training data for robot learning.
• To improve the robustness and generalization of robot policies and success detectors in out-of-distribution (OOD) environments with novel objects and cluttered scenes.
• To develop a general-purpose data augmentation framework that preserves semantic and spatial consistency while enabling zero-shot adaptation to new tasks.

Proposed method

• Leverage state-of-the-art text-to-image diffusion models (e.g., DALL-E 2, Stable Diffusion) to perform semantic inpainting on existing robot demonstration images.
• Parse natural language instructions to identify regions to modify—such as target objects, distractors, or backgrounds—using prompt engineering.
• Apply inpainting with text guidance to replace specific image regions while preserving the rest of the scene, including robot pose and object grasp.
• Generate diverse, realistic training data by systematically varying object identities, colors, and scene contexts using prompt variations.
• Train downstream models (e.g., policies, success detectors) on the augmented dataset, combining real data with synthetic, semantically consistent data.
• Use CLIP-based success detectors to evaluate generalization on OOD test sets with unseen distractors and novel objects.

Experimental results

Research questions

• RQ1 Can text-guided image generation models produce realistic, semantically meaningful data augmentations that improve zero-shot generalization in robot learning?
• RQ2 To what extent can ROSIE enhance policy generalization to novel objects and unseen environments without additional real-world data collection?
• RQ3 Does ROSIE improve the robustness of high-level tasks like success detection in out-of-distribution (OOD) scenarios with novel distractors?
• RQ4 How does the scale of augmentation using diffusion models affect downstream performance on both in-distribution and OOD benchmarks?

Key findings

• ROSIE significantly improves the F1 score of success detection on OOD test sets—from 0.19 with no augmentation to 0.57 with full ROSIE augmentation—demonstrating strong robustness to novel distractors.
• The performance on in-distribution tasks remains stable (F1 ≈ 0.66), indicating that ROSIE does not degrade performance on seen distributions.
• Policies trained on ROSIE-augmented data successfully generalize to completely unseen tasks, such as moving a yellow chip bag near a peach, even when the objects were not in the original training data.
• The method enables effective data augmentation that preserves spatial and semantic consistency, allowing the model to learn task-relevant features despite visual changes.
• ROSIE achieves improved generalization without requiring simulation or motion data, relying solely on image-level semantic augmentation.
• Temporal consistency is preserved in policy learning despite frame-wise augmentation, as shown by stable performance on the Robotics Transformer architecture.