[Paper Review] Dykstra splitting and an approximate proximal point algorithm for minimizing the sum of convex functions

C. H. Jeffrey Pang|arXiv (Cornell University)|Sep 27, 2017

Optimization and Variational Analysis31 references3 citations

TL;DR

This paper extends Dykstra's splitting algorithm to minimize the sum of convex functions by incorporating a regularizing quadratic term, proving convergence to the primal minimizer under relaxed conditions—allowing non-cyclic updates and multiple function minimization. It further introduces an approximate proximal point algorithm using Dykstra's splitting, achieving O(1/n) convergence when a dual minimizer exists.

ABSTRACT

We show that Dykstra's splitting for projecting onto the intersection of convex sets can be extended to minimize the sum of convex functions and a regularizing quadratic. We give conditions for which convergence to the primal minimizer holds so that more than one convex function can be minimized at a time, the convex functions are not necessarily sampled in a cyclic manner, and the SHQP strategy for problems involving the intersection of more than one convex set can be applied. When the sum does not involve the regularizing quadratic, we discuss an approximate proximal point method combined with Dykstra's splitting to minimize this sum.

Motivation & Objective

• To generalize Dykstra’s splitting algorithm from projecting onto intersections of convex sets to minimizing the sum of convex functions with a regularizing quadratic.
• To establish convergence to the primal minimizer without requiring cyclic updates or constraint qualifications.
• To develop an approximate proximal point algorithm that leverages Dykstra’s splitting for minimizing sums of convex functions without the regularizing quadratic.
• To enable the use of the SHQP (supporting halfspace and quadratic programming) heuristic for accelerating convergence in problems with multiple convex sets.
• To provide conditions under which the algorithm achieves O(1/n) convergence rate when a dual minimizer exists.

Proposed method

• Adapts Dykstra’s splitting to the problem of minimizing ∑ᵢ hᵢ(x) + ½‖x − x₀‖² by replacing projections with proximal mappings for general convex functions.
• Uses block coordinate minimization on the dual problem, treating each function hᵢ as a separate block, with updates not required to be cyclic.
• Introduces a supporting halfspace (SHQP) heuristic to approximate the intersection of convex sets more accurately than individual sets, improving convergence.
• Applies the proximal point algorithm by solving a sequence of subproblems of the form (1.3) using Dykstra’s splitting, with adaptive center updates.
• Employs Fenchel duality and support functions to derive dual conditions and ensure convergence via the squeeze theorem on distance to the optimal solution.
• Uses compactness and error bound arguments to verify that the iterates approach the intersection of convex sets, satisfying feasibility conditions.

Experimental results

Research questions

• RQ1 Can Dykstra’s splitting be extended to minimize sums of convex functions with a regularizing quadratic, beyond just projections onto sets?
• RQ2 Under what conditions does Dykstra’s splitting converge to the primal minimizer when the functions are minimized in a non-cyclic order?
• RQ3 Can the SHQP heuristic be formally justified and integrated into Dykstra’s splitting to improve convergence for problems involving multiple convex sets?
• RQ4 What convergence rate can be guaranteed for the approximate proximal point algorithm when combined with Dykstra’s splitting?
• RQ5 How can feasibility and proximity to the optimal solution be verified in practice, especially when the intersection of sets is not known exactly?

Key findings

• Dykstra’s splitting converges to the primal minimizer for minimizing ∑ᵢ hᵢ(x) + ½‖x − x₀‖² without constraint qualifications, even when updates are non-cyclic.
• The algorithm achieves O(1/n) convergence rate when a dual minimizer exists, as shown via the squeeze theorem applied to the distance from iterates to the optimal solution.
• The SHQP heuristic improves convergence by constructing tighter halfspace approximations of the intersection of convex sets using subgradients from previous iterates.
• An approximate proximal point algorithm is proposed that uses Dykstra’s splitting to solve subproblems, with convergence guaranteed under mild conditions.
• Feasibility of the iterates is verified using distance estimates to individual sets, and compactness ensures that small distances to each set imply small distance to their intersection.
• The method allows for simultaneous minimization of more than one convex function and does not require the functions to be sampled in a fixed cyclic order.