[Paper Review] What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

Ajay Mandlekar, Danfei Xu+8 more|arXiv (Cornell University)|Aug 6, 2021

Reinforcement Learning in RoboticsComputer Science84 references70 citations

TL;DR

This paper conducts a comprehensive offline-learning study on robot manipulation using human demonstrations, comparing six algorithms across multiple tasks and data qualities, and provides practical insights for observation spaces, history-dependence, and dataset size.

ABSTRACT

Imitating human demonstrations is a promising approach to endow robots with various manipulation capabilities. While recent advances have been made in imitation learning and batch (offline) reinforcement learning, a lack of open-source human datasets and reproducible learning methods make assessing the state of the field difficult. In this paper, we conduct an extensive study of six offline learning algorithms for robot manipulation on five simulated and three real-world multi-stage manipulation tasks of varying complexity, and with datasets of varying quality. Our study analyzes the most critical challenges when learning from offline human data for manipulation. Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation. We also highlight opportunities for learning from human datasets, such as the ability to learn proficient policies on challenging, multi-stage tasks beyond the scope of current reinforcement learning methods, and the ability to easily scale to natural, real-world manipulation scenarios where only raw sensory signals are available. We have open-sourced our datasets and all algorithm implementations to facilitate future research and fair comparisons in learning from human demonstration data. Codebase, datasets, trained models, and more available at https://arise-initiative.github.io/robomimic-web/

Motivation & Objective

• Assess the challenges of learning from offline human demonstrations in robotic manipulation.
• Compare six offline learning algorithms across simulated and real tasks with datasets of varying quality.
• Identify design choices that critically impact performance (history, observation space, hyperparameters).
• Provide practical guidelines and open-source datasets/code to enable reproducible research.

Proposed method

• Evaluate six algorithms: Behavioral Cloning (BC), BC with RNN (BC-RNN), Hierarchical BC (HBC), BCQ, Conservative Q-Learning (CQL), and IRIS.
• Use five simulated and three real-world multi-stage manipulation tasks.
• Collect datasets from Machine-Generated, Proficient-Human, and Multi-Human sources with low-dim and image observation spaces.
• Train policies with binary task rewards and evaluate checkpoints online to identify best-performing policies.
• Analyze effects of observation spaces, history, dataset size, and hyperparameters.
• Provide open-source datasets, code, and trained models for fair comparisons.

Experimental results

Research questions

• RQ1 How do history-dependent models perform compared to static policies when learning from human demonstrations?
• RQ2 How does data quality (single vs multi-human) affect offline learning performance?
• RQ3 What is the impact of observation space (low-dim vs image) on policy learning from human data?
• RQ4 How do dataset size and hyperparameters influence offline learning for manipulation tasks?
• RQ5 Can findings transfer from simulation to real-world robotic tasks?

Key findings

• History-dependent models (BC-RNN, HBC, IRIS) outperform non-temporal baselines on human datasets, especially for longer-horizon tasks and multi-human data.
• Batch RL methods (BCQ, CQL) excel on machine-generated data but struggle with human demonstrations.
• Observation space and hyperparameters substantially affect performance; including relevant proprioceptive signals can help, while unnecessary ones can hurt, and pixel randomization plus wrist camera observations improve visuomotor learning.
• Larger, high-quality human datasets enable proficient policies on complex tasks; results from simulation transfer to real-world tasks with careful observation and training choices.
• Model selection in offline RL is non-trivial; evaluating policies online in simulation shows the best policy can differ from validation-loss or final-checkpoint choices.
• Wrist-mounted camera observations and image randomization are important for real-world visuomotor imitation.