[Paper Review] On the linear convergence rates of exchange and continuous methods for total variation minimization

Axel Flinth, Frédéric de Gournay+1 more|arXiv (Cornell University)|Jun 24, 2019

Sparse and Compressive Sensing TechniquesEngineering31 references32 citations

TL;DR

The paper analyzes an exchange algorithm for total-variation regularized inverse problems over Radon measures, proving eventual linear convergence under regularity and showing linear convergence for continuously optimizing masses and locations, plus an alternating scheme combining both approaches.

ABSTRACT

We analyze an exchange algorithm for the numerical solution total-variation regularized inverse problems over the space M($\\Omega$) of Radon measures on a subset $\\Omega$ of R d. Our main result states that under some regularity conditions, the method eventually converges linearly. Additionally, we prove that continuously optimizing the amplitudes of positions of the target measure will succeed at a linear rate with a good initialization. Finally, we propose to combine the two approaches into an alternating method and discuss the comparative advantages of this approach.

Motivation & Objective

• Motivate and analyze iterative algorithms for infinite-dimensional total variation minimization over Radon measures.
• Establish convergence rates, including eventual linear convergence for exchange algorithms under regularity assumptions.
• Show linear convergence for gradient-based continuous optimization of Dirac masses’ amplitudes and locations.
• Propose and discuss an alternating exchange-continuous strategy combining the strengths of both approaches.
• Illustrate the results in 1D and 2D total-variation problems.

Proposed method

• Formulate the infinite-dimensional problem as inf_{μ∈M(Ω)} ||μ||_M + f(Aμ) with A: M(Ω) → R^m and f convex.
• Use an exchange algorithm that discretizes Ω with a grid Ω_k and updates Ω_{k+1} = Ω_k ∪ X_k where X_k consists of local maximizers of |A^* q_k| exceeding 1.
• Leverage duality between the primal and its dual, with strong duality under stated assumptions, to relate μ^*, q^* via A^* q^* ∈ ∂||μ^*||_M and -q^* ∈ ∂ f(A μ^*).
• Provide a generic convergence result: under differentiability of f with Lipschitz gradient (or a suitable alternative), a subsequence converges to a primal-dual solution; uniqueness yields whole-sequence convergence.
• Introduce Assumptions 5–6 to handle non-degenerate source conditions, and derive linear convergence rates for the exchange scheme when the dual certificate is non-degenerate.
• Present auxiliary bounds and lemmas connecting discretization distance to dual variables and objective gaps.

Experimental results

Research questions

• RQ1 Does an exchange algorithm for total variation minimization over Radon measures converge to a solution of the infinite-dimensional problem?
• RQ2 Under non-degenerate dual certificates, does the exchange algorithm achieve linear convergence and how does the discretization distance affect convergence?
• RQ3 Can a gradient-based continuous optimization of Dirac masses’ locations and amplitudes achieve linear convergence to the true sparse measure?
• RQ4 Does an alternating exchange-continuous method inherit convergence guarantees and yield practical efficiency?
• RQ5 How do the results extend to 1D and 2D total-variation regularized problems in signal processing contexts?

Key findings

• The exchange algorithm converges (subsequence) to a primal-dual solution under mild assumptions; the objective value converges as well.
• Under a non-degenerate source condition for the dual certificate, the set of active support points in X_k stabilizes to s points after finite iterations, and the cost contracts linearly.
• A well-initialized gradient descent on the pair (α, x) converges linearly to μ^*, with an explicit basin of attraction described.
• Alternating between exchange steps and continuous optimization can combine global convergence guarantees with the efficiency of first-order methods.
• The results are illustrated for total variation minimization in 1D and 2D, with emphasis on quadratic fidelity terms.
• The analysis connects to semi-infinite programming via the dual problem and demonstrates how dual certificates govern convergence behavior.