[Paper Review] Learning Dense Rewards for Contact-Rich Manipulation Tasks

Zheng Wu, Wenzhao Lian+3 more|arXiv (Cornell University)|Nov 17, 2020

Reinforcement Learning in Robotics39 references8 citations

TL;DR

This paper proposes DREM, a method to learn dense reward functions for contact-rich manipulation tasks using multi-modal observations (e.g., images and tactile feedback) without adversarial training. By leveraging self-supervised backward sampling from a single expert demonstration, DREM learns a latent task-progress representation that enables sample-efficient reinforcement learning, outperforming baselines in peg-in-hole and USB insertion tasks.

ABSTRACT

Rewards play a crucial role in reinforcement learning. To arrive at the desired policy, the design of a suitable reward function often requires significant domain expertise as well as trial-and-error. Here, we aim to minimize the effort involved in designing reward functions for contact-rich manipulation tasks. In particular, we provide an approach capable of extracting dense reward functions algorithmically from robots' high-dimensional observations, such as images and tactile feedback. In contrast to state-of-the-art high-dimensional reward learning methodologies, our approach does not leverage adversarial training, and is thus less prone to the associated training instabilities. Instead, our approach learns rewards by estimating task progress in a self-supervised manner. We demonstrate the effectiveness and efficiency of our approach on two contact-rich manipulation tasks, namely, peg-in-hole and USB insertion. The experimental results indicate that the policies trained with the learned reward function achieves better performance and faster convergence compared to the baselines.

Motivation & Objective

• To reduce the manual effort required in designing dense reward functions for contact-rich robotic manipulation tasks.
• To enable dense reward learning from high-dimensional, multi-modal observations (e.g., vision and tactile feedback) without relying on adversarial training.
• To develop a self-supervised data collection method that generates training data using only a single expert demonstration.
• To improve sample efficiency and convergence speed in reinforcement learning by learning a task-progress proxy from observations.
• To enable seamless integration of learned rewards into standard RL algorithms for complex manipulation tasks.

Proposed method

• DREM uses a backward sampling strategy to generate a dataset of observations paired with estimated task progress, starting from the goal state.
• It trains an encoder-decoder network to map high-dimensional observations to a latent space representing task progress, serving as a dense reward signal.
• The method avoids adversarial training by relying on self-supervision and a single expert trajectory to define progress labels.
• Task progress is modeled as a continuous, monotonic proxy for distance to the goal, learned via contrastive learning in the latent space.
• The learned reward function is used with standard RL algorithms like Soft Actor-Critic to train policies.
• The approach is designed to be scalable to high-dimensional, continuous observation spaces common in vision and tactile sensing.

Experimental results

Research questions

• RQ1 Can dense reward functions be learned from high-dimensional, multi-modal observations without adversarial training?
• RQ2 Can a single expert demonstration suffice to generate effective dense rewards through self-supervised data collection?
• RQ3 Does learning a latent task-progress representation improve sample efficiency and convergence in contact-rich manipulation tasks?
• RQ4 How does the performance of DREM compare to handcrafted and adversarially trained reward functions in real-world robotic tasks?
• RQ5 Can the learned reward function capture task-specific concepts such as 'in-contact' or 'losing-contact'?

Key findings

• Policies trained with DREM's learned reward achieved faster convergence and better performance than those trained with handcrafted or adversarially learned rewards on peg-in-hole and USB insertion tasks.
• The method required only one expert demonstration to generate effective dense rewards, significantly reducing data and annotation cost.
• DREM's self-supervised sampling process efficiently generated diverse training data without requiring large-scale expert trajectories.
• The learned reward function demonstrated robustness to high-dimensional observations, including images and tactile feedback, enabling effective policy learning.
• The latent space learned by DREM captured meaningful task-progress dynamics, including physical contact states such as 'in-contact' and 'losing-contact'.
• Experiments showed that DREM outperformed state-of-the-art reward learning baselines in both final task success rate and training sample efficiency.