[Paper Review] Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks

Bo-Han Wu, Suraj Nair+2 more|arXiv (Cornell University)|Jun 19, 2021

Reinforcement Learning in Robotics1 citations

TL;DR

EMBR is a model-based reinforcement learning method that learns a repertoire of visuomotor primitive skills from raw images, using a learned dynamics model, critic, and success classifier to enable robust, sequential execution of long-horizon tasks. It achieves 85% success rate on complex desk organization tasks requiring up to 12 skills and 14 distinct primitives, generalizing to novel objects.

ABSTRACT

In this paper, we study the problem of learning a repertoire of low-level skills from raw images that can be sequenced to complete long-horizon visuomotor tasks. Reinforcement learning (RL) is a promising approach for acquiring short-horizon skills autonomously. However, the focus of RL algorithms has largely been on the success of those individual skills, more so than learning and grounding a large repertoire of skills that can be sequenced to complete extended multi-stage tasks. The latter demands robustness and persistence, as errors in skills can compound over time, and may require the robot to have a number of primitive skills in its repertoire, rather than just one. To this end, we introduce EMBR, a model-based RL method for learning primitive skills that are suitable for completing long-horizon visuomotor tasks. EMBR learns and plans using a learned model, critic, and success classifier, where the success classifier serves both as a reward function for RL and as a grounding mechanism to continuously detect if the robot should retry a skill when unsuccessful or under perturbations. Further, the learned model is task-agnostic and trained using data from all skills, enabling the robot to efficiently learn a number of distinct primitives. These visuomotor primitive skills and their associated pre- and post-conditions can then be directly combined with off-the-shelf symbolic planners to complete long-horizon tasks. On a Franka Emika robot arm, we find that EMBR enables the robot to complete three long-horizon visuomotor tasks at 85% success rate, such as organizing an office desk, a file cabinet, and drawers, which require sequencing up to 12 skills, involve 14 unique learned primitives, and demand generalization to novel objects.

Motivation & Objective

• To address the challenge of learning a robust, reusable repertoire of low-level visuomotor skills for long-horizon tasks.
• To enable skill sequencing that is resilient to errors and environmental perturbations through continuous success monitoring.
• To train a task-agnostic dynamics model using data from all skills, improving sample efficiency and generalization.
• To integrate learned skills with symbolic planners for high-level task execution in real-world environments.
• To achieve high success rates on complex, multi-stage visuomotor tasks involving novel objects and dynamic conditions.

Proposed method

• EMBR employs a learned dynamics model to plan and predict outcomes of skill execution in a model-based RL framework.
• A critic network evaluates the quality of skill policies, providing policy improvement signals during training.
• A success classifier acts as a dense reward function and continuously monitors whether a skill has succeeded or failed, enabling retry mechanisms.
• The success classifier grounds skills in real-world conditions, allowing the system to detect failure and trigger re-attempts under perturbations.
• The dynamics model is trained on a diverse dataset spanning all skills, making it task-agnostic and reusable across different long-horizon tasks.
• Learned visuomotor primitives with their pre- and post-conditions are composed with off-the-shelf symbolic planners to solve complex, multi-stage tasks.

Experimental results

Research questions

• RQ1 Can a model-based RL framework learn a diverse repertoire of visuomotor skills from raw images that generalize to unseen objects?
• RQ2 How can skill success be continuously monitored and re-attempted in the presence of perturbations or failures?
• RQ3 Can a single, task-agnostic dynamics model be effectively trained on data from multiple skills to improve sample efficiency?
• RQ4 To what extent can learned skills be composed with symbolic planners to solve long-horizon, multi-stage tasks?
• RQ5 What is the achievable success rate of such a system on complex, real-world visuomotor tasks involving up to 12 sequential skills?

Key findings

• EMBR achieved an 85% success rate on three long-horizon visuomotor tasks involving desk organization, file cabinet management, and drawer manipulation.
• The system successfully sequenced up to 12 distinct skills across tasks, demonstrating scalability in skill composition.
• The method required 14 unique learned visuomotor primitives, each grounded via a success classifier for robust execution.
• The task-agnostic dynamics model enabled efficient learning across diverse skills using shared data.
• The framework generalized to novel objects not seen during training, indicating strong generalization capabilities.
• The integration of learned skills with symbolic planners enabled reliable execution of complex, multi-stage procedures.