[Paper Review] Differentiating through Log-Log Convex Programs

Akshay Agrawal, Stephen Boyd|arXiv (Cornell University)|Apr 27, 2020

Advanced Optimization Algorithms Research49 references4 citations

TL;DR

This paper presents an efficient method to differentiate through log-log convex programs (LLCPs), enabling gradient computation for parametrized LLCPs via automatic differentiation. By leveraging convex optimization differentiation techniques and a log-log transformation, the approach allows training of differentiable optimization layers in PyTorch and TensorFlow, demonstrated in applications like monotonic regression and queuing system design with state-of-the-art performance.

ABSTRACT

We show how to efficiently compute the derivative (when it exists) of the solution map of log-log convex programs (LLCPs). These are nonconvex, nonsmooth optimization problems with positive variables that become convex when the variables, objective functions, and constraint functions are replaced with their logs. We focus specifically on LLCPs generated by disciplined geometric programming, a grammar consisting of a set of atomic functions with known log-log curvature and a composition rule for combining them. We represent a parametrized LLCP as the composition of a smooth transformation of parameters, a convex optimization problem, and an exponential transformation of the convex optimization problem's solution. The derivative of this composition can be computed efficiently, using recently developed methods for differentiating through convex optimization problems. We implement our method in CVXPY, a Python-embedded modeling language and rewriting system for convex optimization. In just a few lines of code, a user can specify a parametrized LLCP, solve it, and evaluate the derivative or its adjoint at a vector. This makes it possible to conduct sensitivity analyses of solutions, given perturbations to the parameters, and to compute the gradient of a function of the solution with respect to the parameters. We use the adjoint of the derivative to implement differentiable log-log convex optimization layers in PyTorch and TensorFlow. Finally, we present applications to designing queuing systems and fitting structured prediction models.

Motivation & Objective

• To enable efficient gradient computation through parametrized log-log convex programs (LLCPs), which are nonconvex but transform to convex problems under logarithmic transformation.
• To support sensitivity analysis and parameter optimization in LLCPs by computing derivatives of the solution map with respect to problem parameters.
• To integrate differentiable LLCPs into deep learning frameworks like PyTorch and TensorFlow via CVXPY-based layers.
• To demonstrate practical utility in applications such as monotonic regression and queuing system design with structured constraints.
• To provide a scalable, end-to-end method for training models where the prediction is the solution of an LLCP with learnable parameters.

Proposed method

• Transform the parametrized LLCP into a convex optimization problem via the log-log transformation $ u = \log x $, $ F(u) = \log f(e^u) $, converting log-log convex functions into convex ones.
• Represent the solution map as a composition of: (1) smooth parameter transformation, (2) a convex optimization problem, and (3) an exponential transformation of the solution.
• Use recently developed methods for differentiating through convex optimization problems to compute the derivative of the full composition efficiently.
• Implement the method in CVXPY, enabling users to specify parametrized LLCPs and compute gradients or adjoints in just a few lines of code.
• Leverage the adjoint of the derivative to build differentiable optimization layers compatible with PyTorch and TensorFlow.
• Apply the method to train LLCP-based models by backpropagating through the solution map using the adjoint gradient.

Experimental results

Research questions

• RQ1 Can the solution map of a log-log convex program be differentiated efficiently with respect to its parameters?
• RQ2 How can the derivative of the solution to a parametrized LLCP be computed without explicit symbolic differentiation?
• RQ3 Can differentiable LLCP layers be integrated into deep learning frameworks for end-to-end training?
• RQ4 What performance gains can be achieved in structured prediction tasks by enforcing monotonicity through LLCPs?
• RQ5 How does the differentiable LLCP approach compare to standard least-squares fitting in terms of monotonicity and generalization?

Key findings

• The method enables efficient computation of the gradient of a scalar function of the LLCP solution with respect to its parameters, using the adjoint of the solution map’s derivative.
• The implementation in CVXPY allows users to define and differentiate parametrized LLCPs with minimal code, supporting both forward and adjoint gradients.
• In a monotonic regression task, the LLCP-based model achieved a validation error of 0.0077 after 10 training iterations, improving from 0.0081 with a least-squares initialization.
• The LLCP model produced monotonic predictions, unlike the non-monotonic least-squares fit, which is a key advantage for structured prediction.
• Each training iteration, involving 150 LLCP solves and differentiations, took approximately 10 seconds on a 2012 MacBook Pro, demonstrating practical scalability.
• The approach successfully enables differentiable optimization layers in PyTorch and TensorFlow, enabling end-to-end training of models with embedded optimization.