[Paper Review] Computer-Assisted Design of Accelerated Composite Optimization Methods: OptISTA

Uijeong Jang, Shuvomoy Das Gupta+1 more|arXiv (Cornell University)|May 25, 2023

Advanced Optimization Algorithms Research4 citations

TL;DR

This paper introduces OptISTA, a novel first-order optimization method that achieves a factor-of-2 acceleration over FISTA for composite convex optimization problems. By developing a double-function stepsize-optimization performance estimation problem (PEP) as a nonconvex QCQP, the authors design OptISTA through spatial branch-and-bound, and establish its exact optimality via a novel extension of the semi-interpolated zero-chain construction to composite setups.

ABSTRACT

The accelerated composite optimization method FISTA (Beck, Teboulle 2009) is suboptimal by a constant factor, and we present a new method OptISTA that improves FISTA by a constant factor of 2. The performance estimation problem (PEP) has recently been introduced as a new computer-assisted paradigm for designing optimal first-order methods. In this work, we present a double-function stepsize-optimization PEP methodology that poses the optimization over fixed-step first-order methods for composite optimization as a finite-dimensional nonconvex QCQP, which can be practically solved through spatial branch-and-bound algorithms, and use it to design the exact optimal method OptISTA for the composite optimization setup. We then establish the exact optimality of OptISTA under the large-scale assumption with a lower-bound construction that extends the semi-interpolated zero-chain construction (Drori, Taylor 2022) to the double-function setup of composite optimization. By establishing exact optimality, our work concludes the search for the fastest first-order methods, with respect to the performance measure of worst-case function value suboptimality, for the proximal, projected-gradient, and proximal-gradient setups involving a smooth convex function and a closed proper convex function.

Motivation & Objective

• To close the gap in convergence rate between unconstrained and composite first-order methods by designing an optimal algorithm for composite problems.
• To extend the performance estimation problem (PEP) framework—previously limited to single-function, unconstrained settings—to handle composite optimization with two functions.
• To develop a practical computational method for solving the resulting nonconvex QCQP formulation of optimal stepsize selection in fixed-step first-order methods.
• To establish exact optimality of the proposed method by constructing a novel lower bound that matches the upper bound of OptISTA precisely.
• To resolve the open problem of whether FISTA or projected-gradient methods could be improved by a factor of 2, akin to OGM in the unconstrained case.

Proposed method

• Formulate the design of optimal fixed-step first-order methods for composite convex optimization as a finite-dimensional nonconvex quadratically constrained quadratic program (QCQP), using a double-function PEP framework.
• Apply spatial branch-and-bound algorithms to solve the resulting nonconvex QCQP, enabling practical computation of optimal stepsize sequences.
• Introduce a novel proximal zero-respecting sequence construction to model the behavior of proximal methods under the PEP framework.
• Extend the semi-interpolated zero-chain lower-bound construction to the double-function setting of composite optimization, enabling exact optimality certification.
• Use orthogonal matrix embedding to reduce general composite problems to zero-respecting sequences, preserving the structure needed for lower-bound analysis.
• Prove that the optimal method OptISTA achieves the best possible convergence rate by matching the upper and lower bounds exactly.

Experimental results

Research questions

• RQ1 Can the PEP methodology be extended from single-function unconstrained optimization to composite optimization problems involving two functions?
• RQ2 Is it possible to design a first-order method for composite problems that achieves a factor-of-2 acceleration over FISTA, similar to OGM in the unconstrained case?
• RQ3 Can a novel lower-bound construction be developed to exactly match the upper bound of a new optimal method in the composite setting?
• RQ4 Does the proposed method OptISTA achieve the fastest possible convergence rate for proximal-gradient and projected-gradient setups?
• RQ5 Can the performance estimation problem be adapted to handle the non-smooth and non-convex structure inherent in composite optimization?

Key findings

• OptISTA achieves a convergence rate that is exactly a factor of 2 faster than FISTA for composite convex optimization problems.
• The proposed double-function PEP formulation successfully models fixed-step first-order methods for composite problems as a nonconvex QCQP, solvable via spatial branch-and-bound.
• A new lower-bound construction based on the semi-interpolated zero-chain is extended to the double-function composite setting, enabling exact optimality certification.
• The exact match between the upper bound (OptISTA) and the lower bound proves that OptISTA is the fastest possible first-order method for the proximal-gradient and projected-gradient setups.
• The work resolves the long-standing open problem of whether FISTA could be improved by a factor of 2, confirming that such a speedup is both achievable and optimal.
• The methodology demonstrates that computer-assisted design via PEP can be successfully extended beyond unconstrained, single-function problems to general composite optimization.