[Paper Review] Model-free Learning with Heterogeneous Dynamical Systems: A Federated LQR Approach

Han Wang, Leonardo F. Toso+2 more|arXiv (Cornell University)|Aug 22, 2023

Privacy-Preserving Technologies in DataComputer Science3 citations

TL;DR

This paper proposes FedLQR, a federated, model-free reinforcement learning algorithm for heterogeneous linear time-invariant (LTI) systems, enabling multiple agents with distinct yet similar dynamics to collaboratively learn a common stabilizing policy via policy gradient methods. The key contribution is theoretical guarantees that FedLQR ensures iterative stability across all agents and achieves a sample complexity reduction proportional to the number of agents in low-heterogeneity regimes, accelerating convergence to each agent's optimal policy.

ABSTRACT

We study a model-free federated linear quadratic regulator (LQR) problem where M agents with unknown, distinct yet similar dynamics collaboratively learn an optimal policy to minimize an average quadratic cost while keeping their data private. To exploit the similarity of the agents' dynamics, we propose to use federated learning (FL) to allow the agents to periodically communicate with a central server to train policies by leveraging a larger dataset from all the agents. With this setup, we seek to understand the following questions: (i) Is the learned common policy stabilizing for all agents? (ii) How close is the learned common policy to each agent's own optimal policy? (iii) Can each agent learn its own optimal policy faster by leveraging data from all agents? To answer these questions, we propose a federated and model-free algorithm named FedLQR. Our analysis overcomes numerous technical challenges, such as heterogeneity in the agents' dynamics, multiple local updates, and stability concerns. We show that FedLQR produces a common policy that, at each iteration, is stabilizing for all agents. We provide bounds on the distance between the common policy and each agent's local optimal policy. Furthermore, we prove that when learning each agent's optimal policy, FedLQR achieves a sample complexity reduction proportional to the number of agents M in a low-heterogeneity regime, compared to the single-agent setting.

Motivation & Objective

• To address high sample complexity in model-free reinforcement learning for control systems.
• To enable collaborative policy learning across multiple agents with distinct but similar LTI dynamics without sharing raw data.
• To ensure the learned common policy is stabilizing for all agents despite dynamical heterogeneity.
• To quantify the performance gap between the common policy and each agent's individual optimal policy.
• To analyze sample complexity reduction when learning individual optimal policies via federated collaboration.

Proposed method

• Federated learning is used to aggregate policy gradients from M agents at a central server, enabling collaborative training without sharing raw data.
• A model-free policy gradient method is employed, estimating gradients via trajectory sampling with smoothing to reduce variance.
• The algorithm enforces iterative stability by constraining policy updates within a set of stabilizing controllers, ensuring all agents remain stable at each round.
• Theoretical analysis leverages Lyapunov-like arguments and concentration inequalities to bound estimation errors and heterogeneity effects.
• Heterogeneity is modeled via perturbations in system matrices, and bounds are derived on the deviation of the common policy from each agent’s optimal policy.
• Convergence is proven under low-heterogeneity assumptions, with explicit sample complexity bounds derived for achieving near-optimal performance.

Experimental results

Research questions

• RQ1 Is the common policy learned via federated learning stabilizing for all agents with heterogeneous dynamics?
• RQ2 How close is the common policy to each agent’s individual optimal policy in terms of cost function gap?
• RQ3 Can each agent achieve faster convergence to its own optimal policy by leveraging data from other agents in the federation?
• RQ4 What is the impact of system heterogeneity on the performance and stability of the federated policy?
• RQ5 What sample complexity reduction is achievable for individual agents when learning via federated collaboration?

Key findings

• FedLQR ensures that the common policy remains stabilizing for all agents at every iteration, even under heterogeneous dynamics.
• The distance between the common policy and each agent’s optimal policy is bounded by a term that decays exponentially with the number of training rounds.
• In the low-heterogeneity regime, FedLQR reduces the sample complexity for learning each agent’s optimal policy by a factor proportional to the number of agents M, compared to single-agent training.
• The convergence rate of FedLQR is governed by the condition number of the optimal covariance matrix and the minimum eigenvalue of the control cost matrix.
• The algorithm achieves near-optimal performance with high probability when the trajectory length, smoothing radius, and sample size are chosen appropriately to control estimation and heterogeneity errors.
• The theoretical analysis confirms that federated learning can be safely applied to model-free LQR with heterogeneous systems, provided heterogeneity is bounded.