[Paper Review] First-Order Primal-Dual Method for Nonlinear Convex Cone Programming

Lei Zhao, Daoli Zhu|arXiv (Cornell University)|Dec 31, 2017

Sparse and Compressive Sensing Techniques76 references3 citations

TL;DR

This paper proposes the Variant Auxiliary Problem Principle (VAPP), a first-order primal-dual algorithm for nonlinear convex cone programming (NCCP) with smooth or nonsmooth objectives and constraints. By combining linearization and a variable distance-like function in an augmented Lagrangian framework, VAPP achieves O(1/t) convergence rates for primal suboptimality, feasibility, and dual suboptimality in the general convex case, and O(1/t²) under strong convexity, with verifiable error bound conditions and connections to Forward-Backward Splitting methods.

ABSTRACT

Nonlinear Convex Cone Programming (NCCP) problems are important and have many practical applications. In this paper, we introduces a flexible first-order primal-dual algorithm called the Variant Auxiliary Problem Principle (VAPP) for solving NCCP problems when the objective function and constraints are smooth and may be nonsmooth. Each iteration of VAPP generates a nonlinear approximation to the primal problem of an augmented Lagrangian method. The approximation incorporates both linearization and a variable distance-like function, and then the iterations of VAPP provide one decomposition property for NCCP. Motivated by recent applications in big data analysis, there has been an explosive growth in interest in the convergence rate analysis of parallel computing algorithms for large scale optimization problem. This paper proposes an iteration-based error bound and linear convergence of VAPP. Some verifiable sufficient conditions of this error bound are also discussed. For the general convex case (without error bound), we establish $O(1/t)$ convergence rate for primal suboptimality, feasibility and dual suboptimality. By adaptively setting in parameters at different iterations, we show an $O(1/t^2)$ rate for the strongly convex case. We further present Forward-Backward Splitting (FBS) formulation of VAPP method and establish the connection between VAPP and other primal-dual splitting methods. Finally, we discuss some issues in the implementation of VAPP.

Motivation & Objective

• To develop a flexible first-order primal-dual algorithm for solving nonlinear convex cone programming (NCCP) problems with both smooth and nonsmooth components.
• To establish convergence rates for primal suboptimality, feasibility, and dual suboptimality in the general convex case.
• To achieve an improved O(1/t²) convergence rate under strong convexity via adaptive parameter selection.
• To provide verifiable sufficient conditions for an iteration-based error bound that ensures linear convergence.
• To establish connections between VAPP and other primal-dual splitting methods, particularly Forward-Backward Splitting (FBS).

Proposed method

• VAPP constructs a nonlinear approximation to the augmented Lagrangian using linearization and a variable distance-like function, enabling decomposition of the NCCP problem.
• At each iteration, the algorithm minimizes a regularized subproblem involving the primal variable and dual multiplier, with a generalized distance function D(u, u^k) and a penalty term on the dual variable.
• The method employs a backtracking strategy to ensure the descent property by controlling the parameter ε^k, which is bounded by β / (B_G + B_Ω + γτ²) to maintain convergence.
• A weighted average of iterates, (ū_t, p̄_t), is used to estimate the solution, with convergence rates derived via a descent inequality on the generalized distance function.
• The algorithm is shown to be equivalent to a Forward-Backward Splitting (FBS) formulation under appropriate parameterization, linking it to established primal-dual splitting frameworks.
• Implementation considerations include adaptive parameter setting and the use of a dual feasibility projection operator Π(·) to enforce constraint satisfaction.

Experimental results

Research questions

• RQ1 Can a first-order primal-dual method achieve O(1/t) convergence for NCCP problems with nonsmooth objectives and constraints?
• RQ2 What conditions ensure a linear convergence rate via an iteration-based error bound in the VAPP framework?
• RQ3 How can the VAPP method be adapted to achieve an O(1/t²) convergence rate in the strongly convex case?
• RQ4 What is the relationship between VAPP and existing primal-dual splitting methods such as Forward-Backward Splitting?
• RQ5 What verifiable sufficient conditions guarantee the existence of an error bound that leads to linear convergence?

Key findings

• The VAPP algorithm achieves O(1/t) convergence rates for primal suboptimality, feasibility, and dual suboptimality in the general convex case, under standard assumptions.
• Under strong convexity, adaptive parameter selection yields an improved O(1/t²) convergence rate for the same convergence metrics.
• An iteration-based error bound condition is established, and verifiable sufficient conditions for this bound are provided, enabling linear convergence under appropriate parameter control.
• The VAPP method is shown to be equivalent to a Forward-Backward Splitting (FBS) formulation, linking it to a well-known class of primal-dual splitting algorithms.
• A descent inequality on the generalized distance function D(u, u^k) + (ε^k / 2γ) ||p - p^k||² is proven, which underpins the convergence and rate analysis.
• The algorithm is bounded and globally convergent to the saddle point (u*, p*) of the augmented Lagrangian under Assumptions 1 and 2.