[Paper Review] Non-Stochastic Control with Bandit Feedback

Paula Gradu, John Hallman+1 more|arXiv (Cornell University)|Aug 12, 2020

Advanced Bandit Algorithms Research34 references4 citations

TL;DR

This paper presents the first efficient sublinear regret algorithm for non-stochastic control of linear dynamical systems with bandit feedback, where only scalar loss values are observed. By introducing a novel bandit convex optimization method with memory, the approach handles time-dependent loss functions and achieves Õ(poly(natural-parameters)T^{3/4}) regret for both known and unknown systems.

ABSTRACT

We study the problem of controlling a linear dynamical system with adversarial perturbations where the only feedback available to the controller is the scalar loss, and the loss function itself is unknown. For this problem, with either a known or unknown system, we give an efficient sublinear regret algorithm. The main algorithmic difficulty is the dependence of the loss on past controls. To overcome this issue, we propose an efficient algorithm for the general setting of bandit convex optimization for loss functions with memory, which may be of independent interest.

Motivation & Objective

• Address the challenge of controlling linear dynamical systems under adversarial perturbations and unknown loss functions with only scalar bandit feedback.
• Overcome the difficulty of time-dependent loss functions that depend on past controls, which complicates gradient estimation in zero-order feedback settings.
• Develop an efficient algorithm that achieves sublinear regret in the non-stochastic control framework, even when the system dynamics are unknown.
• Extend existing convex relaxation and system identification techniques to bandit feedback settings, enabling practical deployment in real-world control applications.
• Demonstrate the effectiveness of the proposed method across diverse loss functions and noise models, including correlated and non-i.i.d. disturbances.

Proposed method

• Propose a new algorithm for bandit convex optimization with memory, designed to handle loss functions that depend on past states and controls.
• Introduce an artificial delay mechanism to decouple the time dependency in loss functions, enabling unbiased gradient estimation in the bandit feedback setting.
• Combine convex relaxation techniques from [4] with non-stochastic system identification from [16, 30] to estimate system dynamics and disturbances in real time.
• Use gradient-based optimization with estimated gradients via perturbation-based methods, adapted to account for memory in the loss function.
• Apply the bandit optimization framework to disturbance-action controllers (DAC), which generalize linear dynamical controllers (LDC) by incorporating past disturbances.
• Implement a two-phase learning process: first identify system dynamics using online estimation (via algorithm 3 or linear regression), then optimize control policies using the estimated model.

Experimental results

Research questions

• RQ1 Can sublinear regret be achieved in non-stochastic control of linear systems when only scalar bandit feedback is available?
• RQ2 How can gradient estimation be made effective in bandit settings where loss functions depend on past control actions (i.e., have memory)?
• RQ3 What is the performance of the proposed algorithm under adversarial perturbations and unknown system dynamics?
• RQ4 How does the algorithm compare to classical LQR and gradient-based policy methods (e.g., GPC) in non-i.i.d. noise environments?
• RQ5 Can the regret bound be maintained when the system dynamics are not known a priori and must be estimated online?

Key findings

• The proposed algorithm achieves Õ(poly(natural-parameters)T^{3/4}) regret for both known and unknown linear dynamical systems under adversarial perturbations and bandit feedback.
• The method outperforms the classical Linear Quadratic Regulator (LQR) in correlated noise settings such as sinusoidal and random walk disturbances, where LQR is suboptimal.
• With decaying learning rates, the algorithm converges to the LQR solution under i.i.d. Gaussian noise, validating its consistency in benign settings.
• The bandit convex optimization with memory framework enables effective learning despite zero-order feedback, overcoming the challenge of time-dependent loss dependencies.
• System identification via algorithm 3 and linear regression both enable stable learning in unknown dynamics settings, with the former showing better performance in the experiments.
• The algorithm maintains strong performance across diverse loss functions, including L2, L1, Linf, and ReLU, demonstrating robustness to non-smooth and non-quadratic costs.