[Paper Review] Adaptive Regret for Control of Time-Varying Dynamics

Paula Gradu, Elad Hazan+1 more|arXiv (Cornell University)|Jul 8, 2020

Advanced Bandit Algorithms Research31 references14 citations

TL;DR

This paper introduces a novel meta-algorithm that transforms any controller with sublinear regret into one with sublinear adaptive regret for time-varying linear dynamical systems. By leveraging online convex optimization with memory and a novel adaptive regret bound, the method enables controllers to dynamically adapt to changing system dynamics, achieving nearly tight regret bounds in non-stochastic environments.

ABSTRACT

We consider the problem of online control of systems with time-varying linear dynamics. This is a general formulation that is motivated by the use of local linearization in control of nonlinear dynamical systems. To state meaningful guarantees over changing environments, we introduce the metric of {\it adaptive regret} to the field of control. This metric, originally studied in online learning, measures performance in terms of regret against the best policy in hindsight on {\it any interval in time}, and thus captures the adaptation of the controller to changing dynamics. Our main contribution is a novel efficient meta-algorithm: it converts a controller with sublinear regret bounds into one with sublinear {\it adaptive regret} bounds in the setting of time-varying linear dynamical systems. The main technical innovation is the first adaptive regret bound for the more general framework of online convex optimization with memory. Furthermore, we give a lower bound showing that our attained adaptive regret bound is nearly tight for this general framework.

Motivation & Objective

• To address the challenge of controlling nonlinear dynamical systems with time-varying dynamics by using local linearization.
• To develop performance guarantees for online control in non-stationary environments where standard regret metrics fail.
• To introduce adaptive regret as a meaningful performance metric in control theory, capturing the ability to adapt to changing dynamics.
• To design an efficient meta-algorithm that converts standard regret-minimizing controllers into adaptive regret-minimizing ones.
• To establish a nearly tight lower bound on adaptive regret for online convex optimization with memory.

Proposed method

• Proposes a meta-algorithm that maintains multiple expert controllers, each updated via online gradient descent on a proxy loss function.
• Uses a hierarchical set system $S_t$ to maintain and update weights over controllers, ensuring logarithmic complexity in time.
• Employs a surrogate proxy cost function to estimate future performance and guide controller selection at each timestep.
• Applies the framework to online convex optimization with memory, deriving the first adaptive regret bound in this general setting.
• Introduces a novel weight update rule that dynamically shifts control to the best-performing expert on any time interval.
• Employs automatic differentiation to compute time-varying linearized system dynamics $(A_t, B_t)$ at each step.

Experimental results

Research questions

• RQ1 Can we design a controller that adapts effectively to time-varying linear dynamics in non-stochastic environments?
• RQ2 How can we extend regret minimization in online control to capture adaptation to changing system behavior?
• RQ3 What is the optimal achievable adaptive regret bound for online convex optimization with memory in time-varying systems?
• RQ4 Can a meta-algorithm efficiently convert a standard regret-minimizing controller into one with sublinear adaptive regret?
• RQ5 Is the derived adaptive regret bound tight up to logarithmic factors?

Key findings

• The proposed meta-algorithm achieves sublinear adaptive regret for time-varying linear dynamical systems, enabling effective adaptation to changing dynamics.
• The method establishes the first adaptive regret bound for online convex optimization with memory, a general framework with broad implications.
• A nearly tight lower bound is proven, showing that the attained adaptive regret bound is optimal up to logarithmic factors.
• Experiments demonstrate that the MARC meta-algorithm outperforms baseline controllers like GPC and iLQR in non-stationary environments with time-varying dynamics.
• The algorithm maintains $O(\log T)$ working sets per timestep, ensuring computational efficiency despite dynamic controller selection.
• Hyperparameter tuning shows that the method is robust, with optimal performance achieved at $\eta = 0.05$ and $\sigma = 10^{-2}$ in experiments.