[Paper Review] Learning real manipulation tasks from virtual demonstrations using LSTM

Rouhollah Rahmatizadeh, Pooya Abolghasemi+2 more|arXiv (Cornell University)|Mar 12, 2016

Robot Manipulation and Learning19 references28 citations

TL;DR

This paper proposes training a robot controller using virtual demonstrations via an LSTM-based network with a Mixture Density Network (MDN) loss to enable transfer to a physical robot. The approach successfully performs complex manipulation tasks on a Baxter robot, outperforms feedforward networks, and learns from imperfect demonstrations to correct errors.

ABSTRACT

Robots assisting the disabled or elderly must perform complex manipulation tasks and must adapt to the home environment and preferences of their user. Learning from demonstration is a promising choice, that would allow the non-technical user to teach the robot different tasks. However, collecting demonstrations in the home environment of a disabled user is time consuming, disruptive to the comfort of the user, and presents safety challenges. It would be desirable to perform the demonstrations in a virtual environment. In this paper we describe a solution to the challenging problem of behavior transfer from virtual demonstration to a physical robot. The virtual demonstrations are used to train a deep neural network based controller, which is using a Long Short Term Memory (LSTM) recurrent neural network to generate trajectories. The training process uses a Mixture Density Network (MDN) to calculate an error signal suitable for the multimodal nature of demonstrations. The controller learned in the virtual environment is transferred to a physical robot (a Rethink Robotics Baxter). An off-the-shelf vision component is used to substitute for geometric knowledge available in the simulation and an inverse kinematics module is used to allow the Baxter to enact the trajectory. Our experimental studies validate the three contributions of the paper: (1) the controller learned from virtual demonstrations can be used to successfully perform the manipulation tasks on a physical robot, (2) the LSTM+MDN architectural choice outperforms other choices, such as the use of feedforward networks and mean-squared error based training signals and (3) allowing imperfect demonstrations in the training set also allows the controller to learn how to correct its manipulation mistakes.

Motivation & Objective

• To enable non-technical users to teach robots complex manipulation tasks via virtual demonstrations in a safe, non-disruptive environment.
• To address the challenge of transferring policies learned in simulation to a physical robot with real-world dynamics and sensor limitations.
• To develop a controller that can handle multimodal, imperfect demonstrations and learn from mistakes during task execution.
• To evaluate the effectiveness of LSTM with MDN loss compared to standard feedforward networks and mean-squared error training.

Proposed method

• Virtual demonstrations of manipulation tasks are collected in a simulated environment to train a deep neural network controller.
• A Long Short-Term Memory (LSTM) recurrent network is used to model temporal dependencies in the demonstration trajectories.
• A Mixture Density Network (MDN) computes a probabilistic error signal to handle the multimodal nature of human demonstrations.
• The trained controller generates trajectories that are mapped to joint angles via an inverse kinematics module for physical execution.
• An off-the-shelf vision system substitutes for geometric knowledge in simulation, enabling real-world perception.
• The policy is transferred to a Rethink Robotics Baxter robot with no retraining on the physical platform.

Experimental results

Research questions

• RQ1 Can a controller trained on virtual demonstrations successfully perform manipulation tasks on a physical robot?
• RQ2 Does the LSTM+MDN architecture outperform feedforward networks with mean-squared error loss in policy learning from demonstrations?
• RQ3 Can the controller learn to correct errors when trained on imperfect demonstrations?
• RQ4 How well does the system generalize from simulation to reality using vision and inverse kinematics?

Key findings

• The controller trained on virtual demonstrations successfully performed manipulation tasks on the physical Baxter robot, demonstrating effective policy transfer.
• The LSTM+MDN architecture significantly outperformed feedforward networks trained with mean-squared error, particularly in handling multimodal demonstration data.
• The system learned to correct manipulation errors when trained on imperfect demonstrations, improving robustness to real-world variations.
• The use of an off-the-shelf vision system and inverse kinematics enabled functional execution on the physical robot without requiring precise simulation fidelity.