[Paper Review] Safe and Adaptive Decision-Making for Optimization of Safety-Critical Systems: The ARTEO Algorithm

Buse Sibel Korkmaz, Marta Zagórowska+1 more|arXiv (Cornell University)|Nov 10, 2022

Advanced Bandit Algorithms Research4 citations

TL;DR

The paper proposes ARTEO, a safe and adaptive algorithm for real-time optimization in safety-critical systems under uncertainty. It formulates multi-armed bandits as constrained optimization using Gaussian processes to model uncertainty, dynamically adjusting exploration via confidence bounds to ensure safety and optimize performance, achieving lower cumulative regret than safe UCB variants in two real-world case studies.

ABSTRACT

We consider the problem of decision-making under uncertainty in an environment with safety constraints. Many business and industrial applications rely on real-time optimization to improve key performance indicators. In the case of unknown characteristics, real-time optimization becomes challenging, particularly because of the satisfaction of safety constraints. We propose the ARTEO algorithm, where we cast multi-armed bandits as a mathematical programming problem subject to safety constraints and learn the unknown characteristics through exploration while optimizing the targets. We quantify the uncertainty in unknown characteristics by using Gaussian processes and incorporate it into the cost function as a contribution which drives exploration. We adaptively control the size of this contribution in accordance with the requirements of the environment. We guarantee the safety of our algorithm with a high probability through confidence bounds constructed under the regularity assumptions of Gaussian processes. We demonstrate the safety and efficiency of our approach with two case studies: optimization of electric motor current and real-time bidding problems. We further evaluate the performance of ARTEO compared to a safe variant of upper confidence bound based algorithms. ARTEO achieves less cumulative regret with accurate and safe decisions.

Motivation & Objective

• To address the challenge of real-time optimization in safety-critical systems where key parameters are unknown and safety constraints must be strictly enforced.
• To develop an algorithm that balances exploration and exploitation adaptively, avoiding unsafe decisions while optimizing performance goals.
• To integrate uncertainty quantification via Gaussian processes into a constrained optimization framework that guarantees safety with high probability.
• To demonstrate the effectiveness of the approach in industrial applications such as electric motor current control and real-time online bidding.
• To outperform existing safe exploration algorithms by maintaining optimization goals while minimizing constraint violations and regret.

Proposed method

• Formulates the multi-armed bandit problem as a constrained mathematical programming problem with safety and performance constraints.
• Uses Gaussian processes to model unknown reward and constraint functions, encoding regularity assumptions through kernel functions.
• Incorporates uncertainty estimates (variance from GP posterior) into the cost function as an adaptive exploration bonus.
• Applies confidence bounds derived from GP regularity assumptions to construct high-probability safety guarantees.
• Adapts the exploration contribution dynamically based on environmental requirements, ensuring safety without a dedicated exploration phase.
• Solves the optimization problem at each step using a constrained solver that respects bid limits, ROI thresholds, and non-negativity constraints.

Experimental results

Research questions

• RQ1 Can we achieve safe and adaptive exploration in safety-critical systems without requiring a separate exploration phase?
• RQ2 How can uncertainty quantification via Gaussian processes be integrated into real-time optimization to balance exploration and exploitation?
• RQ3 To what extent can the algorithm maintain optimization goals (e.g., ROI, current reference) while ensuring safety constraints are never violated?
• RQ4 How does ARTEO compare in performance and safety to safe variants of UCB-based algorithms in real-world applications?
• RQ5 Can the algorithm adapt its exploration behavior based on dynamic environmental requirements?

Key findings

• ARTEO achieved lower cumulative regret than a safe UCB variant in both the electric motor current optimization and online bidding case studies.
• In the online bidding experiment, ARTEO maintained ROI above the safety threshold while setting lower bid prices than the benchmark algorithm.
• The algorithm successfully avoided constraint violations in both case studies, demonstrating high-probability safety guarantees through confidence bounds.
• ARTEO adapted exploration dynamically, avoiding unnecessary or unsafe exploration while still improving performance over time.
• The use of GP-based uncertainty in the cost function enabled effective and safe exploration without requiring a separate exploration phase.
• Simulation results showed that ARTEO achieved higher ROI with lower costs compared to the baseline algorithm in the iPinYou dataset.