[Paper Review] End-to-End Training of Deep Visuomotor Policies

Sergey Levine, Chelsea Finn+2 more|arXiv (Cornell University)|Apr 2, 2015

Reinforcement Learning in Robotics80 references1,390 citations

TL;DR

This paper proposes end-to-end training of deep convolutional neural networks (CNNs) to map raw RGB images directly to robot motor torques, using guided policy search to convert reinforcement learning into supervised learning. The method achieves superior performance and generalization in real-world manipulation tasks—such as screwing caps, inserting blocks, and hanging hangers—by jointly optimizing vision and control, outperforming separate training of perception and control components.

ABSTRACT

Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control. In this paper, we aim to answer the following question: does training the perception and control systems jointly end-to-end provide better performance than training each component separately? To this end, we develop a method that can be used to learn policies that map raw image observations directly to torques at the robot's motors. The policies are represented by deep convolutional neural networks (CNNs) with 92,000 parameters, and are trained using a partially observed guided policy search method, which transforms policy search into supervised learning, with supervision provided by a simple trajectory-centric reinforcement learning method. We evaluate our method on a range of real-world manipulation tasks that require close coordination between vision and control, such as screwing a cap onto a bottle, and present simulated comparisons to a range of prior policy search methods.

Motivation & Objective

• To investigate whether joint end-to-end training of perception and control improves performance in robotic manipulation tasks.
• To reduce reliance on hand-engineered components for perception, state estimation, and low-level control in policy search.
• To enable direct mapping from raw image observations to motor torques using deep neural networks in real-world robotic settings.
• To improve policy generalization and robustness in tasks requiring precise visual coordination and contact dynamics.
• To demonstrate that supervised fine-tuning via guided policy search enables effective training with limited real-world data.

Proposed method

• The policy is represented by a 7-layer CNN with 92,000 parameters, including a novel spatial feature point transformation for improved spatial reasoning.
• Guided policy search transforms policy search into supervised learning by iteratively generating expert demonstrations using a model-free trajectory optimization method.
• The algorithm is formalized as Bregman ADMM (BADMM), ensuring convergence to a locally optimal solution.
• During training, the full state (e.g., object positions) is observed, but at test time, only raw RGB images are used, enabling generalization to novel configurations.
• The method uses stochastic gradient descent for training, with a cost function combining L2 distance, log penalty for precision, and control effort regularization.
• The policy is trained on real-world interaction data with 5-second episodes and 20 Hz control frequency, using a PR2 robot with a fixed camera.

Experimental results

Research questions

• RQ1 Can end-to-end training of vision and control components jointly lead to better performance than training them separately?
• RQ2 Does using raw image observations with deep CNNs enable robust and generalizable visuomotor policies in real-world robotic manipulation?
• RQ3 Can guided policy search effectively convert high-dimensional policy search into supervised learning with limited real-world data?
• RQ4 To what extent does joint training improve generalization to novel object positions and configurations?
• RQ5 How does the proposed method compare to prior policy search methods in terms of success rate and consistency?

Key findings

• The end-to-end trained visuomotor policies achieved higher success rates and better generalization across novel object positions compared to separately trained perception and control components.
• The method successfully learned policies for complex tasks such as screwing caps, inserting blocks into shape sorters, and hanging hangers, all using only tens of minutes of real-world interaction data.
• In simulated comparisons, guided policy search outperformed multiple prior policy search methods when training high-dimensional neural network policies.
• The inclusion of a spatial feature point transformation reduced overfitting and improved spatial reasoning in the CNN policy.
• The cost function with log penalty term enabled precise placement at target locations, especially critical for tasks like bottle cap screwing.
• The policy generalized to unseen configurations without requiring full state information at test time, demonstrating robustness to novel initial conditions.