[Paper Review] Model Predictive Control via On-Policy Imitation Learning

Kwangjun Ahn, Zakaria Mhammedi+3 more|arXiv (Cornell University)|Oct 17, 2022

Advanced Control Systems Optimization4 citations

TL;DR

This paper proposes Forward-Switch, an on-policy imitation learning method that improves data efficiency and performance in learning explicit Model Predictive Control (MPC) policies for constrained linear systems. By leveraging the structure of MPC and switching to LQR after a learned time horizon, the method achieves stable, constraint-satisfying control with suboptimal cost approaching MPC performance, validated through simulations showing superior sample efficiency over behavior cloning and standard forward training.

ABSTRACT

In this paper, we leverage the rapid advances in imitation learning, a topic of intense recent focus in the Reinforcement Learning (RL) literature, to develop new sample complexity results and performance guarantees for data-driven Model Predictive Control (MPC) for constrained linear systems. In its simplest form, imitation learning is an approach that tries to learn an expert policy by querying samples from an expert. Recent approaches to data-driven MPC have used the simplest form of imitation learning known as behavior cloning to learn controllers that mimic the performance of MPC by online sampling of the trajectories of the closed-loop MPC system. Behavior cloning, however, is a method that is known to be data inefficient and suffer from distribution shifts. As an alternative, we develop a variant of the forward training algorithm which is an on-policy imitation learning method proposed by Ross et al. (2010). Our algorithm uses the structure of constrained linear MPC, and our analysis uses the properties of the explicit MPC solution to theoretically bound the number of online MPC trajectories needed to achieve optimal performance. We validate our results through simulations and show that the forward training algorithm is indeed superior to behavior cloning when applied to MPC.

Motivation & Objective

• To address the data inefficiency and distribution shift issues inherent in behavior cloning when imitating MPC controllers.
• To develop a sample-efficient, interactive imitation learning method tailored for constrained linear systems with online MPC feedback.
• To provide theoretical guarantees on stability, constraint satisfaction, and cost suboptimality for the learned controller.
• To improve upon non-interactive imitation learning by enabling on-policy data collection that reduces error compounding.
• To validate the method’s superiority over behavior cloning and standard forward training in simulation.

Proposed method

• Adapt the forward training algorithm—a policy gradient-based on-policy imitation learning method—for MPC applications by modifying it to handle long-horizon and constrained linear systems.
• Introduce a switching mechanism that transitions from MPC to LQR control after a time horizon estimated from data, leveraging the fact that MPC converges to LQR for time-invariant linear systems.
• Use robust MPC as the expert policy to improve the robustness of the imitation process and reduce sensitivity to distribution shift.
• The method collects online trajectories by interacting with the system in feedback with the MPC controller, ensuring that the training distribution matches the deployment distribution.
• Theoretical analysis bounds the number of required MPC trajectories using high-dimensional statistics and statistical learning theory, ensuring convergence to optimal performance.
• The algorithm estimates a time horizon $\hat{\tau}_{\infty}$ after which the system enters a positive invariant set under LQR, enabling safe switching.

Experimental results

Research questions

• RQ1 Can on-policy imitation learning reduce distribution shift and error compounding compared to behavior cloning when learning MPC policies?
• RQ2 What is the theoretical sample complexity required to learn a controller that stabilizes linear systems and satisfies constraints via imitation learning?
• RQ3 How can the forward training algorithm be adapted to handle long-horizon MPC problems with state and input constraints?
• RQ4 Does switching to LQR after a data-estimated time horizon improve sample efficiency and performance in imitation-based MPC?
• RQ5 Can the learned controller achieve cost suboptimality close to MPC with high probability, and what are the theoretical bounds on this suboptimality?

Key findings

• Forward-Switch achieved a mean normalized cost-to-go of approximately 1.034 with only 180 MPC demonstrations on a d=5 system, significantly outperforming standard forward training.
• Standard forward training with 210 demonstrations resulted in a mean normalized cost-to-go of approximately 35, indicating severe performance degradation due to distribution shift.
• The proposed method reduced cost suboptimality to within 3.4% of optimal performance, demonstrating near-optimal control with limited data.
• Forward-Switch outperformed its behavior cloning counterpart even when both used the same 12-step horizon, showing improved sample efficiency and robustness.
• Theoretical analysis provided sample complexity bounds and guarantees on stability and constraint satisfaction, which were absent in prior imitation-based MPC methods.
• The estimated time horizon $\hat{\tau}_{\infty} = 12$ for the d=5 system was sufficient to ensure safe switching to LQR, validating the practicality of the switching strategy.