[Paper Review] Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control

Frederik Ebert, Chelsea Finn+4 more|arXiv (Cornell University)|Dec 3, 2018

Reinforcement Learning in Robotics42 references263 citations

TL;DR

A self-supervised, model-based deep RL approach for vision-based robotic manipulation that learns a predictive video model from unsupervised interaction and uses planning with MPC to achieve diverse, unseen tasks and objects using pixel-based goals, goal images, or goal classifiers.

ABSTRACT

Deep reinforcement learning (RL) algorithms can learn complex robotic skills from raw sensory inputs, but have yet to achieve the kind of broad generalization and applicability demonstrated by deep learning methods in supervised domains. We present a deep RL method that is practical for real-world robotics tasks, such as robotic manipulation, and generalizes effectively to never-before-seen tasks and objects. In these settings, ground truth reward signals are typically unavailable, and we therefore propose a self-supervised model-based approach, where a predictive model learns to directly predict the future from raw sensory readings, such as camera images. At test time, we explore three distinct goal specification methods: designated pixels, where a user specifies desired object manipulation tasks by selecting particular pixels in an image and corresponding goal positions, goal images, where the desired goal state is specified with an image, and image classifiers, which define spaces of goal states. Our deep predictive models are trained using data collected autonomously and continuously by a robot interacting with hundreds of objects, without human supervision. We demonstrate that visual MPC can generalize to never-before-seen objects---both rigid and deformable---and solve a range of user-defined object manipulation tasks using the same model.

Motivation & Objective

• Enable robotic manipulation from raw pixel inputs without external rewards or resets.
• Learn a forward predictive model from unsupervised interaction that generalizes to unseen objects and tasks.
• Provide flexible goal specification methods (pixels, goal images, or classifiers) for planning solutions.
• Demonstrate robustness to clutter, deformable objects, and large perturbations using a single predictive model.

Proposed method

• Train a video-prediction model that predicts future frames conditioned on actions using a transformation-based architecture (DNA/SNA).
• Use model-predictive control with a sampling-based, gradient-free planner to choose actions that minimize a planning cost over a horizon T.
• Employ three planning cost formulations: pixel-distance to a designated goal, registration-based costs with image-to-image alignment to start/goal frames, and classifier-based goals learned via meta-learning (CAML-MAML).
• Apply temporal skip connections (SNA) to handle occlusions and improve multi-step pixel motion prediction.
• Train a registration network jointly with the video predictor using self-supervised flow estimation between randomly sampled frames.
• Discuss trade-offs among cost functions and provide practical guidance for goal specification and robustness.

Experimental results

Research questions

• RQ1 Can a single self-supervised, forward video-prediction model trained from unsupervised robot interaction enable planning-based control for diverse, unseen manipulation tasks?
• RQ2 How do different goal specification mechanisms (pixel-distance, registration, and goal classifiers) affect performance, robustness, and generalization?
• RQ3 Can the approach handle occlusions, clutter, deformable objects, and large perturbations with a single model?
• RQ4 What are effective training and planning strategies to maximize generalization and data efficiency in vision-based robotic control?

Key findings

• A single predictive model trained with unsupervised interaction enables planning-based control for unseen objects and tasks in real robots.
• Pixel-based planning with MPC can drive object manipulation using pixel distance, registered goal images, or classifier-based goals.
• The transformation-based video predictor (DNA/SNA) supports occlusion handling and robust long-horizon predictions for control.
• Registration-based costs improve planning accuracy by aligning current views to start/goal frames, with a self-supervised registration model trained jointly with prediction.
• Classifier-based goals allow abstract task specifications and can be learned from few positive examples via meta-learning (CAML/MAML).
• The approach demonstrates manipulation of rigid and deformable objects, clutter, and large perturbations with a single model and diverse tasks.