[Paper Review] A primal-dual flow for affine constrained convex optimization

Hao Luo|arXiv (Cornell University)|Mar 11, 2021

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper proposes a novel primal-dual dynamical system for affine-constrained convex optimization that ensures exponential convergence via a tailored Lyapunov function. By introducing time-rescaling parameters and a modified dual update with primal velocity feedback, the method achieves global nonergodic linear convergence, outperforming standard saddle-point flows and providing a continuous-time foundation for fast primal-dual algorithms.

ABSTRACT

We introduce a novel primal-dual flow for affine constrained convex optimization problems. As a modification of the standard saddle-point system, our primal-dual flow is proved to possess the exponential decay property, in terms of a tailored Lyapunov function. Then two primal-dual methods are obtained from numerical discretizations of the continuous model, and global nonergodic linear convergence rate is established via a discrete Lyapunov function. Instead of solving the subproblem of the primal variable, we apply the semi-smooth Newton iteration to the subproblem with respect to the multiplier, provided that there are some additional properties such as semi-smoothness and sparsity. Especially, numerical tests on the linearly constrained $l_1$-$l_2$ minimization and the total-variation based image denoising model have been provided.

Motivation & Objective

• To develop a new continuous-time primal-dual dynamical system that ensures exponential convergence for affine-constrained convex optimization.
• To overcome oscillations and instability in standard saddle-point flows by modifying the dual update with a primal velocity term.
• To establish global nonergodic linear convergence rates through a discrete Lyapunov function derived from continuous-time analysis.
• To provide a continuous-time interpretation of the PDHG method, revealing its connection to a modified dynamical system with over-relaxation.
• To enable efficient numerical solution via semi-smooth Newton methods on the dual subproblem when sparsity and semi-smoothness are present.

Proposed method

• Proposes a primal-dual flow system with two time-rescaling parameters γ and β governed by γ′ = μ − γ and β′ = −β, unifying analysis for μ ≥ 0.
• Introduces a modified dual update ∇λℒ(x + x′, λ) instead of ∇λℒ(x, λ), enhancing stability and reducing oscillations.
• Constructs a tailored Lyapunov function E(t) combining primal-dual gap, primal and dual tracking errors, and time-varying weights.
• Proves exponential decay of the Lyapunov function: dE/dt ≤ −E(t), implying E(t) ≤ e⁻ᵗE(0), leading to exponential convergence of primal and dual iterates.
• Derives two primal-dual algorithms via implicit and semi-implicit time discretizations of the continuous flow, ensuring global nonergodic linear convergence.
• Applies semi-smooth Newton iteration to the dual subproblem when f is semi-smooth and sparse, enabling efficient numerical solution.

Experimental results

Research questions

• RQ1 Can a modified primal-dual dynamical system achieve exponential convergence for affine-constrained convex optimization, unlike standard saddle-point flows?
• RQ2 How does the inclusion of the primal velocity term x′ in the dual update affect stability and convergence behavior?
• RQ3 What is the role of time-rescaling parameters γ and β in unifying the analysis for both μ > 0 and μ = 0 cases?
• RQ4 How does the proposed continuous-time model relate to the PDHG algorithm and its over-relaxation mechanism?
• RQ5 Can the dual subproblem be efficiently solved using semi-smooth Newton methods under structural assumptions like sparsity and semi-smoothness?

Key findings

• The proposed primal-dual flow exhibits exponential decay of the tailored Lyapunov function, with dE/dt ≤ −E(t), implying E(t) ≤ e⁻ᵗE(0).
• The primal and dual iterates converge exponentially: |f(x(t)) − f(x*)| + ||Ax(t) − b|| ≤ Ce⁻ᵗ for some constant C.
• Global nonergodic linear convergence rate is established for both implicit and semi-implicit time discretizations via a discrete Lyapunov function.
• The method provides a continuous-time interpretation of the PDHG algorithm, where the term x′ in the dual update corresponds to the over-relaxation x_{k+1} − x_k.
• Numerical experiments confirm the efficiency of the method on l₁-l₂ minimization and total-variation-based image denoising, especially when combined with semi-smooth Newton methods on the dual subproblem.
• The symmetrization of the operator Q in the PPA framework is achieved via the over-relaxation term, explaining the improved convergence behavior of PDHG over standard primal-dual schemes.