[Paper Review] Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learning

Ilnura Usmanova, Yarden As+2 more|arXiv (Cornell University)|Jul 21, 2022

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper proposes LB-SGD, a safe black-box optimization method that uses stochastic gradient descent with adaptive step sizes on a logarithmic barrier approximation to ensure constraint satisfaction during learning. It achieves efficient convergence in high-dimensional non-convex problems with zeroth- and first-order feedback, offering improved sample and computational complexity over existing safe Bayesian optimization methods.

ABSTRACT

Optimizing noisy functions online, when evaluating the objective requires experiments on a deployed system, is a crucial task arising in manufacturing, robotics and many others. Often, constraints on safe inputs are unknown ahead of time, and we only obtain noisy information, indicating how close we are to violating the constraints. Yet, safety must be guaranteed at all times, not only for the final output of the algorithm. We introduce a general approach for seeking a stationary point in high dimensional non-linear stochastic optimization problems in which maintaining safety during learning is crucial. Our approach called LB-SGD is based on applying stochastic gradient descent (SGD) with a carefully chosen adaptive step size to a logarithmic barrier approximation of the original problem. We provide a complete convergence analysis of non-convex, convex, and strongly-convex smooth constrained problems, with first-order and zeroth-order feedback. Our approach yields efficient updates and scales better with dimensionality compared to existing approaches. We empirically compare the sample complexity and the computational cost of our method with existing safe learning approaches. Beyond synthetic benchmarks, we demonstrate the effectiveness of our approach on minimizing constraint violation in policy search tasks in safe reinforcement learning (RL).

Motivation & Objective

• To develop a scalable, safe optimization method for high-dimensional stochastic problems where constraints are unknown and only noisy measurements are available.
• To ensure all optimization iterates remain within the feasible set at all times, preventing unsafe system interactions.
• To achieve better sample and computational complexity than existing safe Bayesian optimization methods in high-dimensional settings.
• To provide a unified framework for both zeroth-order and first-order feedback in safe learning.
• To demonstrate effectiveness in safe reinforcement learning and real-world policy search tasks.

Proposed method

• LB-SGD uses a logarithmic barrier approximation of the constrained optimization problem to enforce feasibility during optimization.
• The method applies stochastic gradient descent with an adaptive step size rule to minimize the log barrier objective.
• For zeroth-order feedback, it employs a zeroth-order gradient estimator based on finite differences with random directions and noise.
• For first-order feedback, it uses noisy gradient measurements directly in the SGD update.
• The algorithm ensures safety by maintaining iterates within the feasible region through the barrier function's structure.
• Convergence is analyzed for non-convex, convex, and strongly convex problems under smoothness and Lipschitz assumptions.

Experimental results

Research questions

• RQ1 Can a safe optimization method be designed that maintains feasibility throughout the learning process without prior knowledge of constraints?
• RQ2 How does the sample and computational complexity of LB-SGD compare to safe Bayesian optimization in high-dimensional settings?
• RQ3 Can LB-SGD achieve convergence to a stationary point in non-convex problems with only noisy function evaluations?
• RQ4 What is the impact of adaptive step size selection on convergence and constraint violation in black-box optimization?
• RQ5 How does LB-SGD perform in safe reinforcement learning policy search compared to existing safe optimization baselines?

Key findings

• LB-SGD achieves convergence to a stationary point in non-convex problems with high probability under smoothness and Lipschitz conditions.
• The method exhibits improved sample complexity scaling compared to safe Bayesian optimization, especially in high dimensions.
• Computational cost scales efficiently with the number of data points and dimensionality, avoiding the cubic scaling of Gaussian process methods.
• Empirical results show significantly lower constraint violations in policy search tasks for safe reinforcement learning compared to baseline methods.
• Theoretical analysis confirms that the log barrier approximation ensures all iterates remain within the feasible set, guaranteeing safety at all times.
• The adaptive step size rule contributes to faster convergence and reduced constraint violation in both synthetic and real-world benchmarks.