[Paper Review] Optimal Covariance Steering for Discrete-Time Linear Stochastic Systems

Fengjiao Liu, George Rapakoulias+1 more|arXiv (Cornell University)|Nov 1, 2022

Mathematical and Theoretical Epidemiology and Ecology Models4 citations

TL;DR

This paper establishes the existence and uniqueness of the optimal control law for steering the state covariance of a discrete-time linear stochastic system to a desired terminal distribution using a quadratic cost function. It proves that the optimal mean and covariance steering problems are separable and shows that the exact solution can be computed via a lossless convex relaxation reformulated as a semi-definite program (SDP), enabling efficient and exact computation of the optimal control law.

ABSTRACT

In this paper, we study the optimal control problem for steering the state covariance of a discrete-time linear stochastic system over a finite time horizon. First, we establish the existence and uniqueness of the optimal control law for a quadratic cost function. Then, we show the separation of the optimal mean and the covariance steering problems. We also develop efficient computational methods to solve for the optimal control law, which is identified as the solution to a semi-definite program. The effectiveness of the proposed approach is demonstrated through numerical examples. In the process, we also obtain some novel theoretical results for a matrix Riccati difference equation, which may be of independent interest.

Motivation & Objective

• To establish the existence and uniqueness of the optimal control law for finite-horizon covariance steering in discrete-time linear stochastic systems.
• To demonstrate the separation of optimal mean and covariance steering problems under quadratic cost.
• To develop a computationally efficient method for solving the optimal control problem via convex optimization.
• To show that a lossless convex relaxation of the non-convex covariance steering problem yields the exact optimal solution through semi-definite programming (SDP).
• To provide theoretical insights into the properties of matrix Riccati difference equations arising in the solution process.

Proposed method

• Formulate the finite-horizon covariance steering problem as an optimal control problem with a quadratic cost function over a discrete-time linear stochastic system with process noise.
• Prove existence and uniqueness of the optimal control law using analysis of a matrix Riccati difference equation derived from the Hamilton-Jacobi-Bellman equation.
• Decouple the optimal mean and covariance steering problems by showing independence in the absence of constraints.
• Reformulate the original non-convex optimization problem as a convex semi-definite program (SDP) via a lossless convex relaxation technique.
• Use strong duality and strict feasibility to prove that the optimal solution to the SDP corresponds exactly to the solution of the original non-convex problem.
• Implement and compare two solution methods: Newton’s method for direct nonlinear optimization and SDP-based convex optimization using YALMIP and MOSEK.

(a) Newton’s method.

Experimental results

Research questions

• RQ1 Does an optimal control law exist for steering the state covariance of a discrete-time linear stochastic system to a given terminal covariance over a finite horizon?
• RQ2 Is the optimal control law unique under a quadratic cost function?
• RQ3 Can the optimal mean and covariance steering problems be solved independently?
• RQ4 Can the non-convex covariance steering problem be exactly solved via convex relaxation?
• RQ5 What are the structural and analytical properties of the matrix Riccati difference equation that arises in the optimal control solution?

Key findings

• The optimal control law for finite-horizon covariance steering in discrete-time linear stochastic systems exists and is unique under the given quadratic cost function.
• The optimal mean and covariance steering problems are separable when no chance constraints are imposed, allowing independent optimization.
• The original non-convex covariance steering problem admits a lossless convex relaxation, and the optimal solution to the resulting semi-definite program (SDP) is also the optimal solution to the original problem.
• The proposed SDP formulation enables exact and efficient computation of the optimal control law using standard convex optimization solvers, with demonstrated scalability over Newton’s method for higher-dimensional systems.
• Numerical results confirm that the SDP-based solution and Newton’s method yield identical optimal trajectories, mean paths, and covariance ellipses, validating the theoretical claims.
• The paper derives novel theoretical properties of the matrix Riccati difference equation that governs the optimal control, which may be of independent interest in control theory.

(b) SDP with lossless convex relaxation.