[Paper Review] Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator

Maryam Fazel, Rong Ge+2 more|arXiv (Cornell University)|Jan 15, 2018

Advanced Control Systems Optimization240 citations

TL;DR

The paper proves that both model-based and model-free policy gradient methods for the infinite-horizon Linear Quadratic Regulator (LQR) globally converge to the optimal policy, with polynomial-time sample and computational complexity, and shows that natural policy gradient improves convergence rate.

ABSTRACT

Direct policy gradient methods for reinforcement learning and continuous control problems are a popular approach for a variety of reasons: 1) they are easy to implement without explicit knowledge of the underlying model 2) they are an "end-to-end" approach, directly optimizing the performance metric of interest 3) they inherently allow for richly parameterized policies. A notable drawback is that even in the most basic continuous control problem (that of linear quadratic regulators), these methods must solve a non-convex optimization problem, where little is understood about their efficiency from both computational and statistical perspectives. In contrast, system identification and model based planning in optimal control theory have a much more solid theoretical footing, where much is known with regards to their computational and statistical properties. This work bridges this gap showing that (model free) policy gradient methods globally converge to the optimal solution and are efficient (polynomially so in relevant problem dependent quantities) with regards to their sample and computational complexities.

Motivation & Objective

• Bridge the gap between reinforcement learning and classical optimal control by establishing global convergence guarantees for policy gradient methods in the LQR setting.
• Show that both exact and model-free (zeroth-order) policy gradient approaches converge to the optimal policy with polynomial sample and computational complexity.
• Demonstrate that natural policy gradient methods have improved convergence rates over naive gradient methods in this non-convex setting.

Proposed method

• Formulate the infinite-horizon LQR with x_{t+1}=Ax_t+Bu_t and quadratic cost with positive definite Q and R.
• Express the cost C(K) when policies are linear: u_t=-Kx_t, with P_K solving its Lyapunov-like equation and C(K)=E_{x0}[x0^T P_K x0].
• Derive the policy gradient ∇C(K)=2E_K Σ_K where E_K=((R+B^T P_K B)K - B^T P_K A) and Σ_K is the state correlation matrix.
• Analyze the non-convex optimization landscape via gradient domination and almost-smoothness to show global convergence properties despite non-convexity.
• Prove global convergence for three exact update rules: (i) gradient descent, (ii) natural policy gradient, (iii) Gauss-Newton, with explicit iteration/complexity bounds.
• Extend to model-free setting using zeroth-order estimation of the gradient and Σ_K via randomized perturbations and rollouts, proving polynomial sample complexity for convergence.
• Provide high-level proof strategy showing that with enough rollout length and accurate estimates, gradient-based updates converge to the optimum even when gradients are estimated from samples.

Experimental results

Research questions

• RQ1 Does policy gradient on the LQR objective converge to the global optimum despite non-convexity?
• RQ2 Can model-free, sample-based policy gradient methods achieve global optimality with polynomial-time guarantees?
• RQ3 How does natural policy gradient compare to standard gradient methods in convergence rate for LQR?
• RQ4 What are the required conditions (e.g., stability of initial policy, data distribution) for the guarantees to hold in the model-free setting?
• RQ5 Can Gauss-Newton-type updates yield stronger convergence results in this framework?

Key findings

• Exact gradient methods achieve global convergence to the optimal policy with provable rates under suitable stepsizes.
• Model-free (zeroth-order) policy gradient and natural policy gradient attain global optima with polynomial computational and sample complexity under stated assumptions.
• Natural policy gradient provides a provably faster convergence rate than naive gradient descent in this LQR setting.
• Gauss-Newton updates offer the strongest theoretical convergence guarantees among the examined methods.
• The analysis combines optimal control theory, first/zeroth-order optimization, and sample-based reinforcement learning to bridge model-based and model-free approaches.