[Paper Review] Online Optimal Control with Affine Constraints

Yingying Li, Subhro Das+1 more|arXiv (Cornell University)|Oct 10, 2020

Advanced Bandit Algorithms Research47 references4 citations

TL;DR

This paper proposes Online Gradient Descent with Buffer Zones (OGD-BZ), a novel online optimal control algorithm for linear systems with affine state and action constraints under bounded disturbances. OGD-BZ ensures robust constraint satisfaction for all time steps and achieves a policy regret upper bound of $\tilde{O}(\sqrt{T})$, making it the first algorithm with theoretical guarantees on both sublinear regret and hard constraint satisfaction under adversarial cost variations.

ABSTRACT

This paper considers online optimal control with affine constraints on the states and actions under linear dynamics with bounded random disturbances. The system dynamics and constraints are assumed to be known and time-invariant but the convex stage cost functions change adversarially. To solve this problem, we propose Online Gradient Descent with Buffer Zones (OGD-BZ). Theoretically, we show that OGD-BZ with proper parameters can guarantee the system to satisfy all the constraints despite any admissible disturbances. Further, we investigate the policy regret of OGD-BZ, which compares OGD-BZ's performance with the performance of the optimal linear policy in hindsight. We show that OGD-BZ can achieve a policy regret upper bound that is the square root of the horizon length multiplied by some logarithmic terms of the horizon length under proper algorithm parameters.

Motivation & Objective

• To design an online control algorithm that simultaneously minimizes adversarially varying convex costs and guarantees hard constraint satisfaction despite bounded random disturbances.
• To address the gap in existing literature by providing theoretical guarantees on both sublinear policy regret and robust constraint satisfaction in time-varying, constrained control settings.
• To explicitly characterize the trade-off between constraint robustness and regret performance through the design of buffer zones in the optimization framework.

Proposed method

• Converts the constrained online optimal control problem into an equivalent online convex optimization (OCO) problem with temporal-coupled costs and constraints using techniques from robust optimization and unconstrained online control.
• Introduces buffer zones to tighten the constraint set and account for approximation errors introduced during problem conversion, ensuring robust feasibility under disturbances.
• Applies the Online Gradient Descent (OGD) algorithm to the transformed OCO problem, resulting in the OGD-BZ algorithm.
• Uses a reformulation of inequality constraints involving absolute values to handle affine constraints in the state and action variables.
• Employs a parameterized search space that balances conservatism (via buffer zone size) and regret performance.
• Derives theoretical bounds on the Frobenius norm of key matrices and gradient norms to establish regret and feasibility guarantees.

Experimental results

Research questions

• RQ1 Can an online control algorithm achieve sublinear policy regret while guaranteeing hard constraint satisfaction under bounded, adversarial disturbances?
• RQ2 How can the trade-off between constraint robustness and regret performance be formally characterized and controlled in online control with affine constraints?
• RQ3 What is the optimal design of buffer zones that ensures constraint satisfaction without overly restricting the feasible action space?

Key findings

• OGD-BZ ensures that all state and action constraints are satisfied for all time steps, regardless of disturbances bounded by $\bar{w}$, when buffer zone parameters are properly chosen.
• The policy regret of OGD-BZ is bounded by $\tilde{O}(\sqrt{T})$, which is sublinear in the horizon length $T$, under appropriate algorithm parameters and assumptions.
• The regret bound depends on system dimensions and disturbance bounds, with $L\delta_1$ remaining linear in $\bar{w}$ despite $\delta_1$'s inverse dependence on $\bar{w}$, due to the quadratic scaling of the gradient bound $G_f$.
• The algorithm explicitly trades off constraint robustness and regret performance: larger buffer zones improve constraint satisfaction at the cost of increased regret, and vice versa.
• Theoretical analysis shows that the feasible set of OGD-BZ is contained within a relaxed constraint set $\Gamma_\epsilon$, and the minimum constraint violation margin is bounded below by $\epsilon_*$.
• The Frobenius norm of the Hessian-like matrix $\bm{M}$ and the dual variables $\bm{Y}^x, \bm{Y}^u, \bm{Z}$ is bounded by $\delta_1 = \Theta(\sqrt{mn} + \sqrt{k_c})$, which contributes to the regret bound.