[Paper Review] Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulation

Rae Jeong, Yusuf Aytar+6 more|arXiv (Cornell University)|Oct 21, 2019

Domain Adaptation and Few-Shot Learning42 references9 citations

TL;DR

This paper proposes a self-supervised sim-to-real adaptation method for visual robotic manipulation using unlabeled real-world data. By leveraging sequence-based self-supervision through a contrastive forward dynamics (CFD) loss that combines temporal contrastive learning with dynamics modeling, the approach improves state representation transfer from simulation to reality, achieving 62% success in real-world cube stacking with only 5 hours of unlabeled real data—surpassing domain randomization and domain adversarial methods.

ABSTRACT

Collecting and automatically obtaining reward signals from real robotic visual data for the purposes of training reinforcement learning algorithms can be quite challenging and time-consuming. Methods for utilizing unlabeled data can have a huge potential to further accelerate robotic learning. We consider here the problem of performing manipulation tasks from pixels. In such tasks, choosing an appropriate state representation is crucial for planning and control. This is even more relevant with real images where noise, occlusions and resolution affect the accuracy and reliability of state estimation. In this work, we learn a latent state representation implicitly with deep reinforcement learning in simulation, and then adapt it to the real domain using unlabeled real robot data. We propose to do so by optimizing sequence-based self supervised objectives. These exploit the temporal nature of robot experience, and can be common in both the simulated and real domains, without assuming any alignment of underlying states in simulated and unlabeled real images. We propose Contrastive Forward Dynamics loss, which combines dynamics model learning with time-contrastive techniques. The learned state representation that results from our methods can be used to robustly solve a manipulation task in simulation and to successfully transfer the learned skill on a real system. We demonstrate the effectiveness of our approaches by training a vision-based reinforcement learning agent for cube stacking. Agents trained with our method, using only 5 hours of unlabeled real robot data for adaptation, shows a clear improvement over domain randomization, and standard visual domain adaptation techniques for sim-to-real transfer.

Motivation & Objective

• To address the challenge of poor generalization of vision-based reinforcement learning agents trained in simulation when deployed on real robots due to reality sim gaps.
• To improve sim-to-real transfer by leveraging unlabeled real-world robot data without requiring paired or aligned data between simulation and reality.
• To develop a self-supervised representation learning method that exploits the temporal structure of robot experience in both simulated and real environments.
• To enhance state representation robustness for long-horizon manipulation tasks like cube stacking under real-world noise, occlusions, and resolution variations.

Proposed method

• A two-stage training procedure is employed: first pre-training a vision-based policy in simulation using synthetic images and proprioception, then adapting the state representation using unlabeled real robot data.
• The adaptation phase uses a sequence-based self-supervised objective—specifically, the contrastive forward dynamics (CFD) loss—that jointly optimizes for temporal consistency and forward dynamics prediction.
• The CFD loss compares positive and negative contrastive pairs of observation-action sequences, encouraging the network to learn a representation where semantically similar sequences are close in latent space.
• The method jointly optimizes the reinforcement learning objective, behavior cloning (BC) to imitate a state-based agent, and the self-supervised CFD loss, enabling stable and efficient training.
• A shared replay buffer between the state-based and vision-based agents facilitates knowledge transfer, improving sample efficiency and training stability.
• The framework uses modality tuning to ensure effective encoder sharing between simulated and real data during adaptation, avoiding catastrophic forgetting.

Experimental results

Research questions

• RQ1 Can sequence-based self-supervision improve sim-to-real transfer in visual robotic manipulation without requiring paired or aligned data between simulation and reality?
• RQ2 How does combining forward dynamics modeling with time-contrastive learning enhance state representation learning in real-world robotic data?
• RQ3 What is the impact of jointly optimizing reinforcement learning, behavior cloning, and self-supervised objectives on policy performance and training stability?
• RQ4 Does end-to-end self-supervised adaptation outperform a two-stage training procedure with modality tuning for sim-to-real transfer?
• RQ5 To what extent can unlabeled real-world data reduce the need for domain randomization or adversarial domain adaptation in vision-based robotic control?

Key findings

• The proposed two-stage self-supervised adaptation with the CFD objective achieved 62% task success in real-world cube stacking, significantly outperforming domain randomization and domain adversarial neural networks (DANN).
• End-to-end self-supervised adaptation with the TCN loss degraded performance due to insufficient encoder sharing, highlighting the importance of modality tuning in the two-stage training process.
• Joint optimization of the reinforcement learning objective, behavior cloning, and self-supervised CFD loss was essential—optimizing only the self-supervised loss led to catastrophic forgetting and task performance collapse.
• The use of a shared replay buffer between the state-based and vision-based agents improved training speed and stability, especially in complex long-horizon tasks.
• Adding a behavior cloning objective to the vision-based agent significantly accelerated learning and maintained performance even when jointly training with self-supervised objectives.
• Ablation studies confirmed that the CFD loss, which leverages both temporal structure and action-conditioned dynamics, was more effective than simpler self-supervised objectives like TCN or DANN for this task.