[Paper Review] Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling

Yuping Luo, Huazhe Xu+1 more|arXiv (Cornell University)|Jul 12, 2019

Reinforcement Learning in Robotics58 references4 citations

TL;DR

This paper proposes Value Iteration with Negative Sampling (VINS), a method that learns conservatively-extrapolated value functions from expert demonstrations to enable self-correcting policies in imitation learning. By using negative sampling to penalize over-optimistic value extrapolation to unseen states, VINS improves sample efficiency and outperforms prior methods in both behavior cloning and reinforcement learning fine-tuning on sparse-reward robotic control tasks.

ABSTRACT

Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations often suffers from the covariate shift problem, which results in cascading errors of the learned policy. We introduce a notion of conservatively-extrapolated value functions, which provably lead to policies with self-correction. We design an algorithm Value Iteration with Negative Sampling (VINS) that practically learns such value functions with conservative extrapolation. We show that VINS can correct mistakes of the behavioral cloning policy on simulated robotics benchmark tasks. We also propose the algorithm of using VINS to initialize a reinforcement learning algorithm, which is shown to outperform significantly prior works in sample efficiency.

Motivation & Objective

• Address the covariate shift problem in imitation learning, where policies and value functions fail to generalize to unseen states due to distributional shift.
• Overcome the issue of false value function extrapolation that leads to compounding errors in policy execution.
• Develop a theoretically grounded method to learn value functions that favor staying near demonstrated trajectories by assigning lower values to unseen states near demonstrations.
• Improve sample efficiency in reinforcement learning by initializing Q-functions and dynamics models with VINS, reducing the need for extensive warm-up.
• Demonstrate that VINS can correct behavioral cloning errors and outperform prior methods in sample efficiency on benchmark robotic control tasks.

Proposed method

• Introduce the concept of conservatively-extrapolated value functions, which assign lower values to states near demonstrations but not in the demonstration set, to encourage policy self-correction.
• Design the VINS algorithm that learns such value functions via negative sampling, where negative samples are generated by perturbing demonstration states with Gaussian noise.
• Formulate a loss function combining supervised learning on demonstration states and a negative sampling loss that penalizes high values at perturbed states not in the demonstration set.
• Learn a dynamics model using supervised learning on demonstration trajectories to predict next states, enabling action selection via value maximization.
• Use the learned value function and dynamics model to initialize a reinforcement learning algorithm, reducing the need for extensive Q-function warm-up.
• Apply a search over actions near the behavioral cloning policy's actions to improve robustness when the dynamics model is inaccurate far from demonstration data.

Experimental results

Research questions

• RQ1 Can we design a value function that extrapolates conservatively to unseen states, thereby enabling self-correcting policies in imitation learning?
• RQ2 How can negative sampling be used to learn value functions that avoid over-optimistic predictions in regions far from demonstration data?
• RQ3 To what extent does initializing a reinforcement learning algorithm with VINS improve sample efficiency compared to prior methods?
• RQ4 What is the contribution of each component—negative sampling, dynamics model, and action search near BC actions—to the overall performance of VINS?
• RQ5 Can the proposed method correct errors made by behavioral cloning policies in sparse-reward robotic control tasks?

Key findings

• VINS significantly improves behavioral cloning performance on simulated robotics tasks, achieving 75.7% success rate on Pick-100 and 55.2% on Push-100, compared to 66.8% and 37.3% for BC alone.
• Ablation studies show that removing negative sampling reduces performance to 48.5% on Pick-100, indicating that negative sampling is crucial for preventing false extrapolation.
• VINS without behavioral cloning supervision performs poorly (28.5% on Pick-100), demonstrating that BC actions are essential for stable policy initialization.
• Using an oracle dynamics model without BC actions still yields 51.4% success on Pick-100, suggesting that dynamics model quality is a key bottleneck.
• When used to initialize RL, VINS outperforms prior state-of-the-art methods like Nair et al. (2018), achieving near-optimal success rates with fewer environment interactions.
• The learning curves show that VINS-initialized RL converges faster and requires fewer samples to reach high performance, especially in early training phases.