[Paper Review] Logarithmic Regret Bound in Partially Observable Linear Dynamical Systems

Sahin Lale, Kamyar Azizzadenesheli+2 more|arXiv (Cornell University)|Mar 25, 2020

Advanced Bandit Algorithms Research61 references50 citations

TL;DR

The paper provides the first finite-time system identification method usable in both open- and closed-loop settings for partially observable linear dynamical systems and introduces AdaptOn, an adaptive online learning algorithm achieving polylogarithmic regret in T steps.

ABSTRACT

We study the problem of system identification and adaptive control in partially observable linear dynamical systems. Adaptive and closed-loop system identification is a challenging problem due to correlations introduced in data collection. In this paper, we present the first model estimation method with finite-time guarantees in both open and closed-loop system identification. Deploying this estimation method, we propose adaptive control online learning (AdaptOn), an efficient reinforcement learning algorithm that adaptively learns the system dynamics and continuously updates its controller through online learning steps. AdaptOn estimates the model dynamics by occasionally solving a linear regression problem through interactions with the environment. Using policy re-parameterization and the estimated model, AdaptOn constructs counterfactual loss functions to be used for updating the controller through online gradient descent. Over time, AdaptOn improves its model estimates and obtains more accurate gradient updates to improve the controller. We show that AdaptOn achieves a regret upper bound of $ ext{polylog}\left(T ight)$, after $T$ time steps of agent-environment interaction. To the best of our knowledge, AdaptOn is the first algorithm that achieves $ ext{polylog}\left(T ight)$ regret in adaptive control of unknown partially observable linear dynamical systems which includes linear quadratic Gaussian (LQG) control.

Motivation & Objective

• Motivate and address finite-time system identification in partially observable LDS.
• Develop a predictor-form estimation method usable in open- and closed-loop settings.
• Propose AdaptOn, an online learning algorithm that updates the controller using counterfactual losses.
• Prove a polylogarithmic regret bound for AdaptOn under strongly convex costs.

Proposed method

• Formulate system in predictor form with Kalman gain F and Abar to enable regression.
• Set up a regularized least squares problem to estimate Markov-parameter-related matrices G_y from input-output data.
• Develop SysId to recover (A,B,C) and the Markov-parameter matrix G(H) via Hankel matrices and Ho-Kalman style steps.
• Define Nature’s y and use b_t(G) to enable counterfactual reasoning for policy evaluation.
• Adopt Disturbance Feedback Control (DFC) with a convex policy parameterization and online gradient updates.
• Operate AdaptOn in epochs with periodic re-estimation and online convex optimization using counterfactual losses.

Experimental results

Research questions

• RQ1 Can model parameters be estimated with finite-time guarantees in closed-loop estimation?
• RQ2 Can an RL algorithm leverage such estimation to achieve significantly reduced regret in partially observable LDS?
• RQ3 How can counterfactual losses be constructed to drive online policy updates in this setting?

Key findings

• Finite-time system identification guarantees: with persistently exciting inputs, estimation errors shrink as tilde-O(1/√T).
• AdaptOn attains a regret upper bound of polylog(T) after T steps under strongly convex losses.
• This work provides the first logarithmic regret result for adaptive control of unknown partially observable linear dynamical systems, including LQG.
• Closed-loop estimation yields improved regret over prior sqrt(T) bounds in related work.
• Corollaries extend the result to approximate optimal LQG controllers when their DFC approximation lies in the considered policy class.