[Paper Review] Online switching control with stability and regret guarantees

Yingying Li, James A. Preiss+4 more|arXiv (Cornell University)|Jan 20, 2023

Advanced Bandit Algorithms ResearchDecision Sciences3 citations

TL;DR

This paper proposes an online switching control algorithm for systems with unknown dynamics and time-varying costs, using a finite pool of candidate controllers—some potentially unstable—while guaranteeing finite-gain stability and sublinear regret relative to the best stabilizing controller. The method combines adaptive switching with stability-aware exploration, ensuring robust performance even when the stabilizing controller is initially unknown.

ABSTRACT

This paper considers online switching control with a finite candidate controller pool, an unknown dynamical system, and unknown cost functions. The candidate controllers can be unstabilizing policies. We only require at least one candidate controller to satisfy certain stability properties, but we do not know which one is stabilizing. We design an online algorithm that guarantees finite-gain stability throughout the duration of its execution. We also provide a sublinear policy regret guarantee compared with the optimal stabilizing candidate controller. Lastly, we numerically test our algorithm on quadrotor planar flights and compare it with a classical switching control algorithm, falsification-based switching, and a classical multi-armed bandit algorithm, Exp3 with batches.

Motivation & Objective

• To address online switching control in systems with unknown nonlinear dynamics and time-varying cost functions.
• To ensure finite-gain stability even when only one of the candidate controllers is stabilizing, and this one is unknown a priori.
• To achieve sublinear policy regret compared to the optimal stabilizing candidate controller.
• To design a practical algorithm that balances exploration for stability and performance optimization under uncertainty.
• To validate the method on real-world dynamics, such as quadrotor planar flight, against classical benchmarks.

Proposed method

• The algorithm uses a finite pool of candidate controllers, including potentially unstabilizing policies, and dynamically switches between them based on real-time feedback.
• It incorporates a stability-aware exploration mechanism that prioritizes controllers likely to maintain system stability, leveraging the assumption that at least one candidate is stabilizing.
• The method employs a regret minimization framework that compares performance against the best stabilizing controller in hindsight.
• A Lyapunov-based stability analysis ensures finite-gain stability throughout the control horizon, even during exploration.
• The algorithm is designed to be adaptive and online, updating controller selection in real time without requiring system model knowledge.
• It integrates principles from online learning and robust control, combining stability guarantees with performance regret bounds.

Experimental results

Research questions

• RQ1 Can an online switching control algorithm guarantee finite-gain stability when only one of the candidate controllers is stabilizing, and this controller is unknown?
• RQ2 Can such an algorithm achieve sublinear regret relative to the optimal stabilizing controller in the presence of unknown dynamics and time-varying costs?
• RQ3 How does the proposed algorithm compare in performance and stability to classical switching control and multi-armed bandit approaches in real-world systems?
• RQ4 What is the trade-off between exploration for stability and performance optimization in online control with unstable candidate policies?
• RQ5 Can the algorithm maintain stability and performance without prior knowledge of the system model or cost functions?

Key findings

• The proposed algorithm guarantees finite-gain stability for the entire control horizon, even when the stabilizing controller is not known in advance.
• The algorithm achieves sublinear policy regret with respect to the optimal stabilizing candidate controller, indicating long-term performance close to the best possible choice.
• Numerical experiments on quadrotor planar flight demonstrate that the algorithm outperforms both falsification-based switching and the Exp3 bandit algorithm in terms of stability and cost minimization.
• The method effectively identifies and leverages stabilizing controllers through adaptive switching, even when initial controllers are unstable.
• The stability guarantee holds under general nonlinear dynamics with process noise, without requiring full system model knowledge.
• The algorithm maintains robustness and performance in dynamic environments with time-varying cost functions.