[Paper Review] Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approach

Yingying Li, Yujie Tang+2 more|arXiv (Cornell University)|Dec 19, 2019

Advanced Control Systems OptimizationEngineering17 citations

TL;DR

This paper proposes ZODPO, a derivative-free distributed reinforcement learning algorithm for decentralized linear quadratic control with partial observations and local costs. By combining zero-order gradient estimation, consensus-based cost averaging, and local policy updates, ZODPO achieves polynomial sample complexity and guarantees stabilizing controllers with high probability, enabling scalable, safe learning in large-scale systems.

ABSTRACT

This paper considers a distributed reinforcement learning problem for decentralized linear quadratic control with partial state observations and local costs. We propose a Zero-Order Distributed Policy Optimization algorithm (ZODPO) that learns linear local controllers in a distributed fashion, leveraging the ideas of policy gradient, zero-order optimization and consensus algorithms. In ZODPO, each agent estimates the global cost by consensus, and then conducts local policy gradient in parallel based on zero-order gradient estimation. ZODPO only requires limited communication and storage even in large-scale systems. Further, we investigate the nonasymptotic performance of ZODPO and show that the sample complexity to approach a stationary point is polynomial with the error tolerance's inverse and the problem dimensions, demonstrating the scalability of ZODPO. We also show that the controllers generated throughout ZODPO are stabilizing controllers with high probability. Lastly, we numerically test ZODPO on multi-zone HVAC systems.

Motivation & Objective

• To develop a distributed reinforcement learning algorithm for decentralized linear quadratic control under partial state observations and local costs.
• To enable scalable learning in large-scale systems with limited communication and storage per agent.
• To provide non-asymptotic performance guarantees, including sample complexity and stability, for distributed policy optimization.
• To handle unbounded Gaussian process noise while ensuring controller stability during learning.
• To numerically validate the optimality and safety of learned controllers on real-world systems like multi-zone HVAC.

Proposed method

• ZODPO uses zero-order optimization to estimate gradients of the global cost with respect to local policy parameters using only cost evaluations on perturbed policies.
• Each agent performs local policy updates based on estimated gradients, leveraging a consensus algorithm to approximate the infinite-horizon global cost via spatial and temporal averaging.
• A distributed perturbation sampling method is designed to generate policy perturbations with limited communication among agents.
• A truncation step is introduced in gradient estimation to bound gradients and ensure stability under unbounded process noise.
• The algorithm uses a static linear policy parameterization $ u_i(t) = K_i x_{\mathcal{I}_i}(t) $, with $ K_i $ updated in parallel across agents.
• The consensus-based cost estimation ensures that all agents collectively estimate the global cost with minimal communication overhead.

Experimental results

Research questions

• RQ1 What is the sample complexity of a distributed, derivative-free policy optimization algorithm for decentralized LQ control?
• RQ2 Can such an algorithm guarantee stabilizing controllers with high probability under unbounded process noise?
• RQ3 How does the performance of the algorithm scale with system dimension and number of agents?
• RQ4 What is the impact of approximate perturbation sampling and cost estimation on convergence?
• RQ5 Can the algorithm be practically applied to real-world systems like multi-zone HVAC with safety and optimality?

Key findings

• The sample complexity of ZODPO to reach an $ \epsilon $-stationary point is $ O(n_K^3 \max(n,N)\epsilon^{-4}) $, showing polynomial scalability with respect to problem dimensions and inverse error tolerance.
• All controllers generated by ZODPO are stabilizing with high probability, even under unbounded Gaussian process noise, due to the truncation step in gradient estimation.
• The algorithm achieves non-asymptotic performance guarantees by bounding errors from approximate perturbation sampling and spatial-temporal cost averaging.
• Numerical experiments on multi-zone HVAC systems demonstrate that ZODPO learns optimal and safe controllers, validating its practical applicability.
• This is the first sample complexity result for distributed learning in decentralized LQ control with partial observations and unbounded noise.
• The theoretical analysis establishes that the expected deviation of the estimated cost from its true value decays as $ O(1/T_J^2) $, ensuring convergence.