[Paper Review] Learning Stabilizing Controllers of Linear Systems via Discount Policy Gradient

Feiran Zhao, Xingyun Fu+1 more|arXiv (Cornell University)|Dec 17, 2021

Model Reduction and Neural Networks4 citations

TL;DR

This paper proposes a model-free discount policy gradient method to stabilize unknown linear systems by adaptively adjusting a discount factor that guides the policy toward the stabilizing region. By leveraging Lyapunov theory and data-driven updates, the method achieves convergence in O(log(1/ε)) trajectories for accuracy ε, significantly reducing sample and simulation costs compared to prior work.

ABSTRACT

Stability is one of the most fundamental requirements for systems synthesis. In this paper, we address the stabilization problem for unknown linear systems via policy gradient (PG) methods. We leverage a key feature of PG for Linear Quadratic Regulator (LQR), i.e., it drives the policy away from the boundary of the unstabilizing region along the descent direction, provided with an initial policy with finite cost. To this end, we discount the LQR cost with a factor, by adaptively increasing which gradient leads the policy to the stabilizing set while maintaining a finite cost. Based on the Lyapunov theory, we design an update rule for the discount factor which can be directly computed from data, rendering our method purely model-free. Compared to recent work \citep{perdomo2021stabilizing}, our algorithm allows the policy to be updated only once for each discount factor. Moreover, the number of sampled trajectories and simulation time for gradient descent is significantly reduced to $\mathcal{O}(\log(1/ε))$ for the desired accuracy $ε$. Finally, we conduct simulations on both small-scale and large-scale examples to show the efficiency of our discount PG method.

Motivation & Objective

• To address the stabilization of unknown linear systems using model-free reinforcement learning.
• To reduce the number of sampled trajectories and simulation time required for convergence in policy gradient methods.
• To ensure the policy remains in the stabilizing region while maintaining finite cost during optimization.
• To develop a purely data-driven update rule for the discount factor using Lyapunov theory.
• To outperform recent methods by enabling single-step policy updates per discount factor value.

Proposed method

• The method introduces a discounted LQR cost function with an adaptive discount factor that guides the policy toward the stabilizing set.
• A Lyapunov-based update rule is derived for the discount factor, allowing direct computation from data without system model knowledge.
• The policy gradient is computed using on-policy rollouts with the discounted cost, ensuring descent toward stabilizing controllers.
• The discount factor is updated iteratively based on a condition that ensures the policy remains in the stabilizing region and maintains finite cost.
• The algorithm requires only one policy update per discount factor value, improving sample efficiency.
• The method is purely model-free, relying solely on observed trajectories for both gradient estimation and discount factor adaptation.

Experimental results

Research questions

• RQ1 Can a model-free policy gradient method stabilize unknown linear systems without requiring system identification?
• RQ2 How can the discount factor be adaptively adjusted to guide the policy toward the stabilizing region while maintaining finite cost?
• RQ3 What is the sample complexity of the proposed method in terms of achieving a desired accuracy ε?
• RQ4 How does the proposed method compare in sample efficiency to prior work such as perdomo2021stabilizing?
• RQ5 Can the discount factor update rule be computed directly from data using Lyapunov theory?

Key findings

• The proposed method achieves convergence in O(log(1/ε)) sampled trajectories for a desired accuracy ε, significantly reducing sample complexity compared to prior approaches.
• The algorithm allows a single policy update per discount factor value, improving computational efficiency.
• The discount factor update rule is derived from Lyapunov theory and can be computed directly from data, ensuring the policy remains in the stabilizing region.
• The method maintains finite cost throughout optimization, avoiding instability during training.
• Simulations on small-scale and large-scale systems confirm the efficiency and robustness of the proposed approach.
• The method outperforms recent work (perdomo2021stabilizing) in terms of sample and simulation efficiency.