[Paper Review] State Alignment-based Imitation Learning

Fangchen Liu, Zhan Ling+2 more|arXiv (Cornell University)|Apr 30, 2020

Reinforcement Learning in RoboticsComputer Science40 references12 citations

TL;DR

This paper proposes a state alignment-based imitation learning method that improves policy imitation when expert and imitator have mismatched dynamics. By aligning state sequences through local and global perspectives via a regularized reinforcement learning objective, the method achieves superior performance in both standard and challenging dynamics-mismatch settings.

ABSTRACT

Consider an imitation learning problem that the imitator and the expert have different dynamics models. Most of existing imitation learning methods fail because they focus on the imitation of actions. We propose a novel state alignment-based imitation learning method to train the imitator by following the state sequences in the expert demonstrations as much as possible. The alignment of states comes from both local and global perspectives. We combine them into a reinforcement learning framework by a regularized policy update objective. We show the superiority of our method on standard imitation learning settings as well as the challenging settings in which the expert and the imitator have different dynamics models.

Motivation & Objective

• Address the challenge of imitation learning when the imitator and expert have different dynamics models, a limitation that undermines most existing action-focused methods.
• Shift focus from action imitation to state sequence alignment to improve policy generalization across dynamic mismatches.
• Develop a unified reinforcement learning framework that integrates local and global state alignment to guide policy training.
• Demonstrate robust performance in standard imitation learning benchmarks and in challenging dynamics-mismatch scenarios.

Proposed method

• Introduce a state alignment mechanism that considers both local (per-step) and global (sequence-level) state correspondence between expert and imitator trajectories.
• Formulate a regularized policy update objective that encourages the imitator to follow expert state sequences, not just actions.
• Integrate state alignment into a reinforcement learning framework using a loss function that penalizes deviations in state trajectories.
• Optimize the policy using a combination of behavioral cloning and reward shaping based on state alignment quality.
• Use dynamic programming or sequence matching techniques to compute optimal state alignments between expert and imitator trajectories.
• Balance the trade-off between imitation fidelity and policy exploration through a regularization term in the policy objective.

Experimental results

Research questions

• RQ1 Can state sequence alignment improve imitation learning performance when expert and imitator dynamics differ?
• RQ2 How does combining local and global state alignment enhance policy generalization in imitation learning?
• RQ3 Does a regularized policy update based on state alignment outperform action-based imitation in dynamics-mismatched environments?
• RQ4 To what extent does the method generalize across standard and challenging imitation learning benchmarks?

Key findings

• The proposed method achieves superior performance compared to baseline imitation learning methods in standard benchmark environments.
• The method demonstrates significant improvement in settings where the imitator and expert have substantially different dynamics models.
• State alignment via both local and global perspectives leads to more stable and accurate policy learning than action-only imitation.
• The regularized policy update objective effectively balances imitation accuracy and policy exploration, reducing divergence in state trajectories.