[Paper Review] Modeling an Augmented Lagrangian for Blackbox Constrained Optimization

Robert B. Gramacy, Genetha A. Gray+5 more|arXiv (Cornell University)|Mar 19, 2014

Advanced Multi-Objective Optimization Algorithms28 references4 citations

TL;DR

This paper proposes a hybrid method combining Gaussian process surrogates with the augmented Lagrangian (AL) framework to solve constrained blackbox optimization problems where constraints are expensive to evaluate. By modeling the AL function via a surrogate and using expected improvement for global search, the approach efficiently navigates complex feasible regions, outperforming traditional methods on a real hydrology benchmark with a nonconvex feasible set.

ABSTRACT

Constrained blackbox optimization is a difficult problem, with most approaches coming from the mathematical programming literature. The statistical literature is sparse, especially in addressing problems with nontrivial constraints. This situation is unfortunate because statistical methods have many attractive properties: global scope, handling noisy objectives, sensitivity analysis, and so forth. To narrow that gap, we propose a combination of response surface modeling, expected improvement, and the augmented Lagrangian numerical optimization framework. This hybrid approach allows the statistical model to think globally and the augmented Lagrangian to act locally. We focus on problems where the constraints are the primary bottleneck, requiring expensive simulation to evaluate and substantial modeling effort to map out. In that context, our hybridization presents a simple yet effective solution that allows existing objective-oriented statistical approaches, like those based on Gaussian process surrogates and expected improvement heuristics, to be applied to the constrained setting with minor modification. This work is motivated by a challenging, real-data benchmark problem from hydrology where, even with a simple linear objective function, learning a nontrivial valid region complicates the search for a global minimum. Supplementary materials for this article are available online.

Motivation & Objective

• Address the lack of statistical methods for constrained blackbox optimization, especially when constraints are expensive to evaluate and nontrivial.
• Overcome the limitation of existing statistical approaches that treat constraints as binary or ignore distance to feasibility boundaries.
• Develop a framework that combines global search via surrogate modeling with local optimization via the augmented Lagrangian to improve convergence and feasibility.
• Enable application of expected improvement heuristics to constrained problems by modeling the augmented Lagrangian as a surrogate objective.
• Demonstrate effectiveness on a real-world hydrology problem with a simple linear objective but highly nonconvex feasible region due to simulation-based constraints.

Proposed method

• Model the augmented Lagrangian function of the constrained optimization problem using a Gaussian process surrogate to enable global exploration.
• Use expected improvement (EI) on the surrogate model of the AL function to guide sequential sampling and balance exploration and exploitation.
• Integrate the AL framework to handle constraints by transforming the constrained problem into an unconstrained one via penalty terms and dual variables.
• Apply trust-region optimization to the AL subproblem using quadratic models and early termination based on gradient norms.
• Use a hybrid strategy: the statistical model explores globally, while the AL framework localizes search near the optimal solution.
• Tolerate small constraint violations (≤10⁻³) in classical comparators to ensure fair performance evaluation, while enforcing strict feasibility in the proposed method.

Experimental results

Research questions

• RQ1 Can a surrogate model of the augmented Lagrangian function improve global convergence in constrained blackbox optimization with expensive constraints?
• RQ2 How does combining expected improvement with the augmented Lagrangian framework compare to classical and statistical methods in terms of feasibility and convergence speed?
• RQ3 To what extent does modeling the AL function rather than the original objective or constraints improve optimization performance on nonconvex feasible regions?
• RQ4 How effective is the hybrid approach on real-world problems with simple objectives but complex, simulation-based constraints?
• RQ5 Does the proposed method maintain strict feasibility while achieving faster convergence than methods that allow constraint violations?

Key findings

• The proposed method successfully navigates a highly nonconvex feasible region in a real hydrology benchmark, finding a global minimum that is difficult to reach with standard approaches.
• The hybrid approach outperforms classical augmented Lagrangian solvers (e.g., NOMAD) and stochastic optimization (SA) in terms of convergence speed and feasibility.
• By modeling the augmented Lagrangian as a surrogate, the method leverages global search capabilities of expected improvement while maintaining local convergence through the AL framework.
• The method achieves strict feasibility (no constraint violations) throughout, unlike classical comparators that allow small violations for fairness, demonstrating robustness.
• The approach is effective even when the objective function is simple (linear), highlighting its value in problems where constraints dominate the difficulty.
• The use of a surrogate model for the AL function enables efficient use of a limited blackbox evaluation budget, particularly in high-dimensional and nonconvex settings.