[Paper Review] TaSIL: Taylor Series Imitation Learning

Daniel Pfrommer, Thomas T. C. K. Zhang+2 more|arXiv (Cornell University)|May 30, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

TaSIL introduces a novel imitation learning method that enhances behavior cloning by penalizing discrepancies in higher-order Taylor series terms between the learned and expert policies. By leveraging incremental input-to-state stability, TaSIL ensures that small errors in Taylor approximations lead to small trajectory deviations, achieving generalization bounds scaling as Õ(1/n) and demonstrating significant performance gains over baselines in MuJoCo environments.

ABSTRACT

We propose Taylor Series Imitation Learning (TaSIL), a simple augmentation to standard behavior cloning losses in the context of continuous control. TaSIL penalizes deviations in the higher-order Taylor series terms between the learned and expert policies. We show that experts satisfying a notion of $ extit{incremental input-to-state stability}$ are easy to learn, in the sense that a small TaSIL-augmented imitation loss over expert trajectories guarantees a small imitation loss over trajectories generated by the learned policy. We provide sample-complexity bounds for TaSIL that scale as $ ilde{\mathcal{O}}(1/n)$ in the realizable setting, for $n$ the number of expert demonstrations. Finally, we demonstrate experimentally the relationship between the robustness of the expert policy and the order of Taylor expansion required in TaSIL, and compare standard Behavior Cloning, DART, and DAgger with TaSIL-loss-augmented variants. In all cases, we show significant improvement over baselines across a variety of MuJoCo tasks.

Motivation & Objective

• To address distribution shift in offline imitation learning by directly encoding expert robustness into the policy via Taylor series terms.
• To provide theoretical guarantees that small TaSIL-augmented loss on expert trajectories implies small imitation error on trajectories generated by the learned policy.
• To develop a finite-difference approximation for TaSIL applicable to experts without access to higher-order derivatives.
• To establish generalization bounds scaling as Õ(1/n) in the realizable setting for n expert demonstrations.
• To empirically validate the theory that more robust experts require lower-order Taylor expansions in TaSIL and to show performance gains over standard BC, DART, and DAgger.

Proposed method

• TaSIL augments standard behavior cloning loss by adding penalties on the differences between the p-th order Taylor series approximations of the expert and learned policies.
• The method relies on the assumption that the expert policy is incrementally input-to-state stable (δ-ISS), which ensures robust recovery from perturbations.
• A finite-difference approximation is used to estimate higher-order derivatives when explicit access to Jacobians or Hessians is unavailable.
• The loss function includes weighted terms for zeroth, first, and second-order Taylor terms, with hyperparameters λ₀, λ₁, λ₂ tuned for performance.
• Theoretical analysis reduces the generalization of TaSIL to a supervised learning problem over expert data, enabling finite-sample bounds.
• Experiments use MuJoCo environments with 4500 training iterations, Adam optimizer, and cosine learning rate decay, comparing TaSIL-augmented BC, DART, and DAgger.

Experimental results

Research questions

• RQ1 Under what conditions does a small TaSIL-augmented imitation loss on expert trajectories guarantee small imitation error on the learned policy’s trajectories?
• RQ2 How does the required order of Taylor expansion in TaSIL relate to the robustness of the expert policy?
• RQ3 Can TaSIL achieve better generalization than standard behavior cloning, DART, and DAgger in offline imitation learning settings?
• RQ4 What is the sample complexity of TaSIL in the realizable setting, and does it scale as Õ(1/n)?
• RQ5 How effective is the finite-difference approximation for estimating higher-order derivatives in practice?

Key findings

• TaSIL achieves generalization bounds scaling as Õ(1/n) in the realizable setting, where n is the number of expert demonstrations.
• The required order of Taylor expansion in TaSIL correlates with expert robustness: more robust experts require lower-order expansions.
• In MuJoCo environments, TaSIL-augmented behavior cloning outperforms standard BC, DART, and DAgger, especially on hard tasks where BC fails.
• For high-dimensional observation spaces (e.g., Humanoid, Ant), using multiple finite difference vectors significantly improves TaSIL performance.
• The finite-difference approximation enables effective TaSIL training even when higher-order derivatives are not directly accessible.
• Empirical results confirm that TaSIL reduces trajectory deviation from the expert policy by explicitly matching higher-order policy dynamics.