[Paper Review] Learning to Optimize Under Constraints with Unsupervised Deep Neural Networks

Seyedrazieh Bayati, Faramarz Jabbarvaziri|arXiv (Cornell University)|Jan 3, 2021

Manufacturing Process and Optimization4 citations

TL;DR

This paper proposes a novel unsupervised deep learning framework that enables real-time constrained optimization by training a deep neural network to map problem parameters to near-optimal solutions while enforcing generic equality and inequality constraints through a piecewise regularization method. The approach achieves near-optimal feasibility and performance comparable to interior-point methods with drastically reduced online inference cost.

ABSTRACT

In this paper, we propose a machine learning (ML) method to learn how to solve a generic constrained continuous optimization problem. To the best of our knowledge, the generic methods that learn to optimize, focus on unconstrained optimization problems and those dealing with constrained problems are not easy-to-generalize. This approach is quite useful in optimization tasks where the problem's parameters constantly change and require resolving the optimization task per parameter update. In such problems, the computational complexity of optimization algorithms such as gradient descent or interior point method preclude near-optimal designs in real-time applications. In this paper, we propose an unsupervised deep learning (DL) solution for solving constrained optimization problems in real-time by relegating the main computation load to offline training phase. This paper's main contribution is proposing a method for enforcing the equality and inequality constraints to the DL-generated solutions for generic optimization tasks.

Motivation & Objective

• Address the gap in existing learn-to-optimize methods that focus only on unconstrained problems, especially in dynamic environments requiring frequent re-optimization.
• Develop a generic, differentiable method to enforce both equality and inequality constraints in deep neural network solutions for continuous optimization.
• Enable real-time inference for optimization tasks where traditional methods like interior-point methods are too slow due to high computational complexity.
• Ensure that the deep learning model generates feasible solutions without relying on post-hoc normalization or heuristic constraint handling.

Proposed method

• Formulate the optimization problem as a deep neural network (DNN) mapping from input parameters $ p $ to optimal solution $ x^* $, with the loss function set to the objective function $ f_0(x;p) $.
• Introduce a piecewise regularization penalty $ oldsymbol{ ext{F}}_i(x^*;p) $ and $ oldsymbol{ ext{H}}_j(x^*;p) $ for inequality and equality constraints, respectively, using a power-based penalty with exponent $ oldsymbol{ ext{γ}} \geq 1 $ and scaling factor $ \boldsymbol{\eta}_i \geq 0 $.
• Replace the non-differentiable indicator function $ \mathbb{I}(\cdot) $ with a continuous, differentiable penalty function that assigns increasing cost to constraint violations.
• Train the DNN using stochastic optimization (e.g., ADAM) to minimize the combined loss: $ \mathcal{L} = f_0(x^*;p) + \sum_i F_i(x^*;p) + \sum_j H_j(x^*;p) $, ensuring gradients propagate through feasible and infeasible regions.
• Use a large number of training samples ($ \sim 1000 $) uniformly distributed across parameter space to generalize across diverse problem setups.
• Enforce constraints implicitly during training, avoiding post-processing or architectural constraints like saturating activations.

Experimental results

Research questions

• RQ1 Can a deep neural network be trained to learn a mapping from problem parameters to optimal solutions under generic equality and inequality constraints in a differentiable, unsupervised manner?
• RQ2 How can constraint violations be penalized in a way that allows gradient-based optimization while ensuring feasibility in the final solution?
• RQ3 To what extent can a DNN-based solution achieve near-optimal performance compared to traditional interior-point methods in constrained optimization?
• RQ4 Can the proposed method generalize across diverse, nonlinear, and non-convex optimization landscapes with varying parameter configurations?

Key findings

• The DNN-generated solutions for Rosenbrock’s function with one constraint closely matched the interior-point method results: for $ c_1=1, c_2=1 $, the DNN achieved $ x_1=0.8394, x_2=0.6040 $, compared to $ 0.8082, 0.5889 $ from the interior-point method.
• For Rosenbrock’s function with three constraints, the DNN successfully generated feasible solutions within the constrained region, even when the global minimum was outside the feasible set.
• In Ackley’s function with one constraint, the DNN produced solutions within $ 10^{-5} $ of the optimal solution (e.g., $ x_1 = -5.6177 \times 10^{-6}, x_2 = 7.2256 \times 10^{-5} $) compared to the interior-point method’s $ \sim 10^{-12} $, demonstrating high accuracy.
• The method maintained feasibility across all test cases, including complex nonlinear constraints, by enforcing penalties that guided the network toward feasible regions during training.
• The computational cost of inference using the DNN was significantly lower than that of interior-point methods, which have worst-case complexity $ \mathcal{O}(\max\{n,m\}^4\sqrt{n}\log(1/\epsilon)) $, enabling real-time deployment.
• With $ \eta_i = 10^8 $ and $ \gamma = 2 $, the penalty terms effectively suppressed infeasible solutions, ensuring that only feasible, near-optimal outputs were learned.