[Paper Review] VILD: Variational Imitation Learning with Diverse-quality Demonstrations

Voot Tangkaratt, Bo Han+2 more|arXiv (Cornell University)|Sep 15, 2019

Reinforcement Learning in Robotics49 references4 citations

TL;DR

VILD proposes a variational imitation learning method that models demonstrator expertise levels using a probabilistic graphical model, enabling robust policy learning from diverse-quality demonstrations without prior knowledge of expert quality. By combining variational inference with importance sampling, VILD achieves state-of-the-art performance on continuous-control benchmarks, demonstrating scalability and data efficiency in realistic IL settings.

ABSTRACT

The goal of imitation learning (IL) is to learn a good policy from high-quality demonstrations. However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations from a mix of experts and amateurs. IL in such situations can be challenging, especially when the level of demonstrators' expertise is unknown. We propose a new IL method called \underline{v}ariational \underline{i}mitation \underline{l}earning with \underline{d}iverse-quality demonstrations (VILD), where we explicitly model the level of demonstrators' expertise with a probabilistic graphical model and estimate it along with a reward function. We show that a naive approach to estimation is not suitable to large state and action spaces, and fix its issues by using a variational approach which can be easily implemented using existing reinforcement learning methods. Experiments on continuous-control benchmarks demonstrate that VILD outperforms state-of-the-art methods. Our work enables scalable and data-efficient IL under more realistic settings than before.

Motivation & Objective

• To address the challenge of imitation learning when demonstrations vary in quality and expert levels are unknown.
• To develop a scalable method that estimates both demonstrator expertise and reward functions simultaneously from low- and high-quality demonstrations.
• To improve data efficiency and robustness in imitation learning by modeling expertise as a latent variable in a probabilistic framework.
• To enable practical deployment of IL in real-world scenarios where high-quality expert demonstrations are scarce or unavailable.

Proposed method

• VILD models the level of demonstrator expertise as a latent variable using a probabilistic graphical model, allowing joint estimation of expertise and reward function.
• It employs a variational inference approach to approximate the intractable posterior over expertise and policy, making the method scalable to large state and action spaces.
• The method uses a variational lower bound (ELBO) to optimize the joint likelihood of demonstrations and estimated expertise levels.
• Importance sampling is applied to reweight trajectories based on estimated expertise, improving data efficiency during policy learning.
• The framework is implemented using standard reinforcement learning algorithms, enabling integration with deep RL methods.
• The reward function is end-to-end trained alongside the policy and expertise estimation, enabling the model to learn from mixed-quality data.

Experimental results

Research questions

• RQ1 Can a probabilistic model effectively estimate the latent expertise level of demonstrators when no ground-truth labels or additional expert signals are available?
• RQ2 How can variational inference be leveraged to scale imitation learning to high-dimensional continuous control tasks with diverse-quality demonstrations?
• RQ3 To what extent does importance sampling based on estimated expertise improve data efficiency in imitation learning?
• RQ4 How does VILD compare to state-of-the-art methods in terms of performance and robustness when trained on mixed-quality demonstrations?
• RQ5 Can the joint estimation of reward function and expertise levels lead to better policy performance than methods that assume fixed expert quality?

Key findings

• VILD outperforms state-of-the-art imitation learning methods on continuous-control benchmarks, even when demonstrations include a significant proportion of low-quality trajectories.
• The method achieves superior performance by effectively identifying and down-weighting low-quality demonstrations through learned expertise estimation.
• VILD demonstrates robustness to noisy and diverse demonstrations, maintaining high policy performance even when expert demonstrations are scarce.
• The integration of variational inference enables scalable training on high-dimensional state and action spaces, overcoming limitations of exact inference.
• Importance sampling based on estimated expertise significantly improves data efficiency, reducing reliance on high-quality demonstrations.
• Empirical results confirm that joint learning of reward and expertise leads to better policy optimization than methods assuming known or fixed expert quality.