[Paper Review] Deep Learning with Experience Ranking Convolutional Neural Network for Robot Manipulator

Van‐Dinh Nguyen, Hung Manh La+1 more|arXiv (Cornell University)|Sep 16, 2018

Reinforcement Learning in Robotics21 references4 citations

TL;DR

This paper proposes an Experience Ranking Convolutional Neural Network (ER-CNN) to enhance DDPG + HER reinforcement learning in robotic manipulation tasks by prioritizing high-reward, high-quality experiences for replay. By using a pre-trained CNN to rank episodes based on visual and task-relevant features, the method accelerates learning convergence and improves final performance, achieving 100% success rate in door pushing and reducing training episodes by up to two-thirds in complex tasks.

ABSTRACT

Supervised learning, more specifically Convolutional Neural Networks (CNN), has surpassed human ability in some visual recognition tasks such as detection of traffic signs, faces and handwritten numbers. On the other hand, even state-of-the-art reinforcement learning (RL) methods have difficulties in environments with sparse and binary rewards. They requires manually shaping reward functions, which might be challenging to come up with. These tasks, however, are trivial to human. One of the reasons that human are better learners in these tasks is that we are embedded with much prior knowledge of the world. These knowledge might be either embedded in our genes or learned from imitation - a type of supervised learning. For that reason, the best way to narrow the gap between machine and human learning ability should be to mimic how we learn so well in various tasks by a combination of RL and supervised learning. Our method, which integrates Deep Deterministic Policy Gradients and Hindsight Experience Replay (RL method specifically dealing with sparse rewards) with an experience ranking CNN, provides a significant speedup over the learning curve on simulated robotics tasks. Experience ranking allows high-reward transitions to be replayed more frequently, and therefore help learn more efficiently. Our proposed approach can also speed up learning in any other tasks that provide additional information for experience ranking.

Motivation & Objective

• Address the challenge of sparse and binary rewards in robotic reinforcement learning, where standard methods struggle due to lack of learning signal.
• Overcome the limitations of fixed replay strategies in Hindsight Experience Replay (HER), which treat all experiences equally regardless of quality.
• Improve sample efficiency and learning speed by introducing a learned, task-specific evaluation mechanism for experience selection.
• Demonstrate that embedding prior knowledge—via a pre-trained CNN—into the learning process can significantly accelerate and stabilize policy learning in robotics tasks.

Proposed method

• Integrate Deep Deterministic Policy Gradient (DDPG) with Hindsight Experience Replay (HER) to handle sparse rewards in robotic manipulation.
• Train a separate, pre-trained Convolutional Neural Network (ER-CNN) to evaluate and rank entire training episodes based on visual and task-relevant features.
• Use the ER-CNN's output score to determine whether an episode should be stored in the replay buffer, with only high-ranking transitions being retained.
• Implement a filtering mechanism that discards low-ranked experiences (e.g., those with ranking >4) to prevent poor-quality data from degrading learning.
• Apply the ranked experience replay to four MuJoCo robotic tasks: door pushing, fetch slide, fetch push, and fetch pick-and-place.
• Use image-based features (e.g., distance to object, door handle presence, hinge angle) as input to the ER-CNN for episode evaluation.

Experimental results

Research questions

• RQ1 Can a learned, external CNN-based ranking system improve the sample efficiency of DDPG + HER in robotic manipulation tasks with sparse rewards?
• RQ2 How does experience ranking based on visual and task-specific features compare to uniform or TD-error-based replay strategies in terms of learning speed and final performance?
• RQ3 To what extent can pre-trained visual models reduce the number of episodes required for convergence in complex robotic control tasks?
• RQ4 Does filtering out low-quality experiences via ER-CNN lead to more stable and higher final success rates compared to standard HER?
• RQ5 Can the proposed method generalize across diverse robotic manipulation tasks beyond the tested environments?

Key findings

• The ER-CNN method achieved 100% median success rate in the door pushing task, significantly outperforming the original DDPG + HER, which failed to maintain high accuracy at convergence.
• In the FetchPush-v1 task, the proposed method required only half the number of episodes to converge compared to the original DDPG + HER.
• For the more complex FetchPick&Place-v1 task, the method reduced the number of required training episodes to one-third of the original DDPG + HER baseline.
• The ER-CNN-based ranking system improved learning robustness, with faster convergence and more stable performance across all four simulated tasks.
• Even when using a real-world trained ER-CNN (80.2% accuracy) instead of a perfect classifier, the method still outperformed the original DDPG + HER in learning speed and final performance.
• The results indicate that prioritizing high-quality, visually informative experiences leads to more efficient and stable policy learning in sparse-reward environments.