[Paper Review] Provable Hierarchical Imitation Learning via EM

Zhe Zhang, Ioannis Ch. Paschalidis|arXiv (Cornell University)|Oct 7, 2020

Robot Manipulation and Learning41 references4 citations

TL;DR

This paper proposes a provably convergent EM-based algorithm for hierarchical imitation learning (HIL), framing HIL as parameter inference in a latent variable model. Under regularity conditions, the algorithm converges with high probability to a norm ball around the true parameters, providing the first theoretical guarantee for HIL using only unsegmented state-action demonstrations without observed options or supervision.

ABSTRACT

Due to recent empirical successes, the options framework for hierarchical reinforcement learning is gaining increasing popularity. Rather than learning from rewards which suffers from the curse of dimensionality, we consider learning an options-type hierarchical policy from expert demonstrations. Such a problem is referred to as hierarchical imitation learning. Converting this problem to parameter inference in a latent variable model, we theoretically characterize the EM approach proposed by Daniel et al. (2016). The population level algorithm is analyzed as an intermediate step, which is nontrivial due to the samples being correlated. If the expert policy can be parameterized by a variant of the options framework, then under regularity conditions, we prove that the proposed algorithm converges with high probability to a norm ball around the true parameter. To our knowledge, this is the first performance guarantee for an hierarchical imitation learning algorithm that only observes primitive state-action pairs.

Motivation & Objective

• To address the lack of theoretical foundations in hierarchical imitation learning (HIL), where existing methods are largely empirical and lack generalization guarantees.
• To formalize HIL as a latent variable model inference problem, enabling rigorous analysis of learning algorithms.
• To establish convergence guarantees for the EM algorithm in HIL, particularly under the challenge of correlated time-series samples.
• To bridge the gap between finite-sample EM performance and population-level convergence, ensuring robustness to initialization.
• To provide theoretical justification for the practical success of EM-based HIL methods in robotics and deep RL.

Proposed method

• Formulate HIL as parameter inference in a latent variable model, where options are unobserved and must be inferred from unsegmented state-action sequences.
• Adopt the Expectation-Maximization (EM) algorithm to jointly estimate the high-level policy and option parameters via iterative E-step and M-step.
• Introduce the population EM algorithm as an intermediate theoretical tool, defined using the infinite-sample limit of the Q-function to ensure convergence to the true parameter.
• Analyze the finite-sample EM algorithm as a perturbation of the population EM, proving high-probability convergence to a norm ball around the true parameter.
• Establish stochastic convergence of the finite-sample Q-function to the population Q-function, even under temporal correlation in the data.
• Use uniform forgetting arguments and concentration inequalities to control the effect of sampling randomness and ensure robustness to initialization.

Experimental results

Research questions

• RQ1 Can the EM algorithm for hierarchical imitation learning be theoretically justified with convergence guarantees?
• RQ2 Does the algorithm converge to a neighborhood of the true parameters when only primitive state-action pairs are observed?
• RQ3 How does the algorithm's performance depend on initialization, sample size, and model complexity?
• RQ4 Can the population EM framework be adapted to time-series latent variable models with dependent samples, such as in HIL?
• RQ5 What is the impact of sampling variability and finite data on the convergence of the EM algorithm in HIL?

Key findings

• Under regularity conditions, the EM algorithm for HIL converges with high probability to a norm ball around the true parameter, providing the first theoretical performance guarantee for HIL with unsegmented demonstrations.
• The population EM algorithm converges to the true parameter, and the finite-sample EM algorithm converges to a neighborhood of the true parameter with high probability, even under correlated time-series data.
• The algorithm's convergence is robust to initialization, with high probability achieving at least a 50% reduction in estimation error within 10 iterations, as shown in empirical validation.
• The effect of the forgetting factor μ on final performance is negligible when sample size T is large, with a maximum performance difference of only 0.7% across μ values for N=1000 iterations.
• Random initialization experiments confirm local convergence: the algorithm succeeds when initial error is moderate (w=0.2), fails when initial error is large (w=0.3), and shows little improvement when already close to the optimum (w=0.1).
• The early-stage exponential decay of estimation error is observed across multiple T values (5000, 8000, 10000), suggesting a general convergence pattern in the initial phase of the algorithm.