[Paper Review] Deep Reinforcement Learning for Robotic Manipulation-The state of the art

Smruti Amarjyoti|arXiv (Cornell University)|Jan 31, 2017

Reinforcement Learning in RoboticsComputer Science20 references57 citations

TL;DR

A survey organizing DRL methods for robotic manipulation by action space (DAS vs CAS) and policy representations (SCAS vs DCAS), detailing key algorithms, architectures, and real-world vs simulated implementations.

ABSTRACT

The focus of this work is to enumerate the various approaches and algorithms that center around application of reinforcement learning in robotic ma- ]]nipulation tasks. Earlier methods utilized specialized policy representations and human demonstrations to constrict the policy. Such methods worked well with continuous state and policy space of robots but failed to come up with generalized policies. Subsequently, high dimensional non-linear function approximators like neural networks have been used to learn policies from scratch. Several novel and recent approaches have also embedded control policy with efficient perceptual representation using deep learning. This has led to the emergence of a new branch of dynamic robot control system called deep r inforcement learning(DRL). This work embodies a survey of the most recent algorithms, architectures and their implementations in simulations and real world robotic platforms. The gamut of DRL architectures are partitioned into two different branches namely, discrete action space algorithms(DAS) and continuous action space algorithms(CAS). Further, the CAS algorithms are divided into stochastic continuous action space(SCAS) and deterministic continuous action space(DCAS) algorithms. Along with elucidating an organ- isation of the DRL algorithms this work also manifests some of the state of the art applications of these approaches in robotic manipulation tasks.

Motivation & Objective

• Motivate the use of DRL for robotic manipulation over traditional hand-engineered policies.
• Organize DRL approaches by discrete vs continuous action spaces and by stochastic vs deterministic policies.
• Explain how deep learning enables end-to-end visuomotor control and policy representations.
• Highlight practical considerations in sim-to-real transfer, training stability, and sample efficiency.

Proposed method

• Classify DRL algorithms into discrete action space (DAS) and continuous action space (CAS).
• Subdivide CAS into stochastic continuous action space (SCAS) and deterministic continuous action space (DCAS).
• Describe core algorithms (DQN, Double DQN, Dueling Networks, NAF, policy gradient variants, TRPO, DDPG) and their applicability to robotics.
• Discuss visuo-motor control with deep networks and experience replay to stabilize learning.
• Summarize implementation aspects including CNN-based policies, actor-critic architectures, and parallel/asynchronous learning.

Experimental results

Research questions

• RQ1 What DRL algorithms and architectures are most effective for robotic manipulation in discrete versus continuous action spaces?
• RQ2 How do policy representations (value-based, policy-based, actor-critic) perform in real-time robotic manipulation tasks?
• RQ3 What are the challenges and solutions for learning from vision inputs and transferring from simulation to real robots?
• RQ4 What methods improve sample efficiency and training stability in DRL for robotics?
• RQ5 What gaps remain in transfer learning and reward specification for complex manipulation tasks?

Key findings

• DAS methods (e.g., DQN variants) are suitable for discrete action robotics tasks but face challenges with continuous action spaces.
• CAS methods (policy search, actor-critic) are more natural for continuous robotic control, with DDPG as a key deterministic policy gradient approach.
• NAF and DDPG show strong performance on continuous control tasks and real-time robotic manipulation like reaching and door opening.
• Experience replay and target networks stabilize DRL training for vision-based robotic control.
• Asynchronous and parallel data collection significantly reduces training time by scaling with the number of robots, improving sample efficiency.
• The survey identifies gaps in transfer learning and reward design, suggesting more work on inverse RL and intrinsic motivation for temporal abstraction.