[Paper Review] A Note on Linear Quadratic Regulator and Kalman Filter

Midhun T. Augustine|arXiv (Cornell University)|Aug 30, 2023

Inertial Sensor and Navigation4 citations

TL;DR

This note provides a theoretical and intuitive introduction to the Linear Quadratic Regulator (LQR) and Kalman Filter (KF), deriving their optimal control and estimation laws for discrete-time linear systems. It presents the derivation of the LQR controller and KF estimator using dynamic programming and Riccati equations, showing convergence to steady-state solutions under observability and controllability, with numerical implementation details and simulation results demonstrating performance across predictor, filter, and smoother variants.

ABSTRACT

Two central problems in modern control theory are the controller design problem: which deals with designing a control law for the dynamical system, and the state estimation problem (observer design problem): which deals with computing an estimate of the states of the dynamical system. The Linear Quadratic Regulator (LQR) and Kalman Filter (KF) solves these problems respectively for linear dynamical systems in an optimal manner, i.e., LQR is an optimal state feedback controller and KF is an optimal state estimator. In this note, we will be discussing the basic concepts, derivation, steady-state analysis, and numerical implementation of the LQR and KF.

Motivation & Objective

• To provide a theoretical foundation for the Linear Quadratic Regulator (LQR) and Kalman Filter (KF) in discrete-time linear systems.
• To derive the optimal control and estimation laws using dynamic programming and Riccati equations.
• To analyze the steady-state behavior of the LQR and KF, including convergence of the Riccati equations and estimator gains.
• To present numerical implementation algorithms for LQR and KF, including predictor, filter, and smoother variants.
• To demonstrate performance differences through simulation, showing that the Kalman smoother yields the lowest estimation variance.

Proposed method

• Derives the LQR controller by minimizing a quadratic cost function using dynamic programming and the Hamilton-Jacobi-Bellman equation.
• Solves the discrete-time Riccati equation for the LQR gain matrix, enabling optimal state feedback control.
• Derives the Kalman filter using the separation principle, minimizing estimation error variance via the Kalman gain and Riccati equation.
• Applies the Kalman predictor, filter, and smoother formulations, with time-varying and steady-state gain computation.
• Uses the Algebraic Riccati Equation (ARE) to analyze steady-state convergence of the estimator under observability and positive definite process noise.
• Employs numerical simulation with an LTI system to compare state estimation performance across predictor, filter, and smoother, using Gaussian noise models.

Figure 1: Response of LTI system with LQR: (a) $N=5,$ (b) $N=50$ .

Experimental results

Research questions

• RQ1 How can the optimal control law for a linear system be derived using quadratic cost minimization?
• RQ2 What conditions ensure convergence of the Kalman filter estimator to a steady-state gain matrix?
• RQ3 How do the performance characteristics of the Kalman predictor, filter, and smoother compare in terms of estimation error variance?
• RQ4 What is the role of the Riccati equation in the derivation and numerical implementation of the LQR and KF?
• RQ5 How does the separation principle enable the design of LQG control using LQR and KF components?

Key findings

• The LQR controller is derived as the solution to a discrete-time optimal control problem, minimizing a quadratic cost function over a finite horizon.
• The Kalman filter estimator minimizes the mean-square estimation error by optimally combining system dynamics and noisy measurements.
• For observable LTI systems with positive definite process noise, the Kalman filter Riccati equation converges to a unique positive definite solution, enabling fixed-gain estimation.
• The Kalman smoother produces the most accurate state estimate with the lowest variance, while the predictor has the highest variance due to lack of future measurements.
• The optimal estimator gain and Riccati matrix converge to time-invariant values in steady state, allowing for reduced online computation after convergence.
• Numerical simulations confirm that the Kalman smoother outperforms the filter and predictor in estimation accuracy, with variance decreasing in the order: predictor > filter > smoother.

Figure 2: Gaussian distribution: graphical representation