[Paper Review] Reinforcement Learning from Imperfect Demonstrations

Yang Gao, Huazhe+4 more|arXiv (Cornell University)|Feb 14, 2018

Reinforcement Learning in Robotics24 references98 citations

TL;DR

NAC unifies learning from demonstrations and environment rewards by normalizing the Q-function, enabling robust learning from imperfect demonstrations and improving beyond the demonstrator’s performance.

ABSTRACT

Robust real-world learning should benefit from both demonstrations and interactions with the environment. Current approaches to learning from demonstration and reward perform supervised learning on expert demonstration data and use reinforcement learning to further improve performance based on the reward received from the environment. These tasks have divergent losses which are difficult to jointly optimize and such methods can be very sensitive to noisy demonstrations. We propose a unified reinforcement learning algorithm, Normalized Actor-Critic (NAC), that effectively normalizes the Q-function, reducing the Q-values of actions unseen in the demonstration data. NAC learns an initial policy network from demonstrations and refines the policy in the environment, surpassing the demonstrator's performance. Crucially, both learning from demonstration and interactive refinement use the same objective, unlike prior approaches that combine distinct supervised and reinforcement losses. This makes NAC robust to suboptimal demonstration data since the method is not forced to mimic all of the examples in the dataset. We show that our unified reinforcement learning algorithm can learn robustly and outperform existing baselines when evaluated on several realistic driving games.

Motivation & Objective

• Motivate robust real-world learning by leveraging both demonstrations and environmental interaction.
• Develop a unified objective that avoids separate supervised and reinforcement losses.
• Enable learning from imperfect or noisy demonstrations without requiring optimality.
• Demonstrate robust performance across demonstrations and environment-based refinements.

Proposed method

• Propose Normalized Actor-Critic (NAC) which normalizes the Q-function to reduce unseen actions from demonstrations.
• Derive NAC updates from the soft policy gradient framework with a unified loss.
• Use a target network and replay buffer to stabilize training, without external imitation losses.
• Incorporate demonstrations into off-policy learning alongside environment transitions via the same objective.
• Show that NAC can learn from imperfect demonstrations and refine policies through interaction.

Experimental results

Research questions

• RQ1 Can NAC learn effectively from both demonstrations and environment-based rewards?
• RQ2 Is NAC robust to suboptimal or noisy demonstrations?
• RQ3 Does NAC outperform imitation-plus-RL baselines on driving-related tasks?
• RQ4 How does NAC compare to existing approaches when demonstrations are limited or noisy?

Key findings

• NAC outperforms prior methods on driving tasks with modest demonstrations and tolerates noisy demonstrations by using rewards rather than pure imitation.
• A unified objective allows learning from demonstrations and environments without auxiliary supervised imitation losses.
• NAC remains robust to imperfect demonstrations and can improve beyond the demonstrator’s performance through environment interaction.
• In toy and realistic driving environments NAC maintains strong performance even with limited demonstration data and varying reward choices.