[Paper Review] MOCCA: mirrored convex/concave optimization for nonconvex composite functions

Rina Foygel Barber, Emil Y. Sidky|arXiv (Cornell University)|Oct 29, 2015

Sparse and Compressive Sensing Techniques43 references17 citations

TL;DR

This paper introduces MOCCA (Mirrored Convex/Concave Optimization), a primal-dual algorithm for nonconvex composite optimization problems of the form $ F(Kx) + G(x) $, where $ F $ and $ G $ may be nonconvex or nondifferentiable. It uses local convex approximations at each iteration and provides convergence guarantees under approximate convexity, with empirical results showing fast convergence on structured signal recovery problems.

ABSTRACT

Many optimization problems arising in high-dimensional statistics decompose naturally into a sum of several terms, where the individual terms are relatively simple but the composite objective function can only be optimized with iterative algorithms. In this paper, we are interested in optimization problems of the form F(Kx) + G(x), where K is a fixed linear transformation, while F and G are functions that may be nonconvex and/or nondifferentiable. In particular, if either of the terms are nonconvex, existing alternating minimization techniques may fail to converge; other types of existing approaches may instead be unable to handle nondifferentiability. We propose the MOCCA (mirrored convex/concave) algorithm, a primal/dual optimization approach that takes local convex approximation to each term at every iteration. Inspired by optimization problems arising in computed tomography (CT) imaging, this algorithm can handle a range of nonconvex composite optimization problems, and offers theoretical guarantees for convergence when the overall problem is approximately convex (that is, any concavity in one term is balanced out by convexity in the other term). Empirical results show fast convergence for several structured signal recovery problems.

Motivation & Objective

• To address nonconvex composite optimization problems where both $ F $ and $ G $ may be nonconvex or nondifferentiable, which existing methods struggle with.
• To develop a method that combines majorization/minimization with primal/dual techniques to handle complex, real-world problems such as image reconstruction in computed tomography (CT).
• To provide theoretical convergence guarantees when the overall problem is approximately convex, i.e., any concavity in one term is balanced by convexity in the other.
• To offer a flexible framework where $ F $ and $ G $ can be locally approximated by convex functions at each iteration, enabling broader applicability.

Proposed method

• Proposes the MOCCA algorithm, a primal-dual method that uses local convex approximations to nonconvex or nondifferentiable functions $ F $ and $ G $ at each iteration.
• Employs a preconditioned Chambolle-Pock-style iterative scheme with extrapolation ($ \theta = 1 $) to solve the saddle-point formulation of the problem.
• At each iteration, replaces $ F $ and $ G $ with convex upper bounds $ F^{(t)} $ and $ G^{(t)} $, forming a convex subproblem that is solved via proximal steps.
• Uses a mirror descent-type update in the dual space, with dual variable $ w $ updated via $ \arg\min_w \{ -\langle K\bar{x}^{t+1}, w \rangle + F^*(w) + \frac{1}{2}\|w - w^t\|^2_{\Sigma^{-1}} \} $.
• Incorporates a stable inner loop version (Algorithm 2) and connects MOCCA to ADMM via a reformulated preconditioned ADMM with convex approximations.
• Employs convergence analysis based on monotonicity and Lyapunov functions, proving sublinear convergence to a solution under approximate convexity.

Experimental results

Research questions

• RQ1 Can a primal-dual algorithm with local convex approximations achieve convergence for nonconvex composite problems where standard methods fail?
• RQ2 Under what conditions does the MOCCA algorithm converge when $ F $ and $ G $ are nonconvex or nondifferentiable?
• RQ3 How does MOCCA compare in performance and robustness to existing alternating minimization or proximal methods on structured signal recovery tasks?
• RQ4 Can the MOCCA framework be connected to ADMM and preconditioned primal-dual methods while preserving convergence guarantees?

Key findings

• MOCCA achieves sublinear convergence to a solution under the condition that the overall problem is approximately convex, i.e., any concavity in $ F $ or $ G $ is balanced by convexity in the other term.
• The algorithm demonstrates fast convergence on structured signal recovery problems, including those arising in computed tomography (CT) imaging.
• Theoretical analysis proves that the iterates converge to a solution with a rate of $ O(1/L) $, where $ L $ is the number of iterations, under appropriate assumptions on the functions and step sizes.
• The convergence guarantee holds even when $ F $ and $ G $ are nondifferentiable or nonconvex, provided they admit local convex approximations with bounded curvature.
• The MOCCA algorithm is shown to be equivalent to a preconditioned ADMM with convex approximations, extending the applicability of ADMM to nonconvex settings.
• Empirical results confirm that MOCCA outperforms standard proximal and alternating minimization methods on nonconvex composite problems, especially in high-dimensional settings.