[Paper Review] On the acceleration of the double smoothing technique for unconstrained convex optimization problems

Radu Ioan Boţ, Christopher Hendrich|arXiv (Cornell University)|May 3, 2012

Sparse and Compressive Sensing Techniques3 references4 citations

TL;DR

This paper accelerates the double smoothing technique for unconstrained convex optimization by analyzing how properties of the objective functions influence convergence rates. It shows that when the functions are strongly convex or have Lipschitz gradients, the convergence rate improves from $O\left(\frac{1}{\epsilon}\ln\left(\frac{1}{\epsilon}\right)\right)$ to $O\left(\frac{1}{\sqrt{\epsilon}}\ln\left(\frac{1}{\epsilon}\right)\right)$ or even $O\left(\ln\left(\frac{1}{\epsilon}\right)\right)$, significantly enhancing efficiency for solving nondifferentiable convex problems via Fenchel duality and fast gradient methods.

ABSTRACT

In this article we investigate the possibilities of accelerating the double smoothing technique when solving unconstrained nondifferentiable convex optimization problems. This approach relies on the regularization in two steps of the Fenchel dual problem associated to the problem to be solved into an optimization problem having a differentiable strongly convex objective function with Lipschitz continuous gradient. The doubly regularized dual problem is then solved via a fast gradient method. The aim of this paper is to show how do the properties of the functions in the objective of the primal problem influence the implementation of the double smoothing approach and its rate of convergence. The theoretical results are applied to linear inverse problems by making use of different regularization functionals.

Motivation & Objective

• To improve the convergence rate of the double smoothing technique for solving unconstrained nondifferentiable convex optimization problems.
• To investigate how the structural properties of the objective functions in the primal problem affect the implementation and convergence speed of the double smoothing approach.
• To establish tighter convergence rates under additional assumptions such as strong convexity or Lipschitz continuous gradients.
• To demonstrate the practical effectiveness of the accelerated method through applications in image deblurring and inverse problems.

Proposed method

• The method employs Fenchel duality to transform the primal problem into a dual problem with a nondifferentiable objective.
• It applies two-stage regularization to the dual problem, making the objective differentiable and strongly convex with a Lipschitz continuous gradient.
• The regularized dual problem is solved using a fast gradient method, which ensures a convergence rate dependent on the smoothness and convexity properties of the original functions.
• An approximately optimal primal solution is reconstructed from the dual iterates using explicit formulas derived from the conjugate functions.
• The approach leverages the biconjugate property and subdifferential calculus to ensure convergence and optimality conditions are met.
• Numerical validation is performed on image deblurring problems using $l_1$ and $l_2$-$l_1$ regularization functionals to compare performance with ISTA and FISTA.

Experimental results

Research questions

• RQ1 How do the convexity and smoothness properties of the functions in the primal problem affect the convergence rate of the double smoothing technique?
• RQ2 Can the convergence rate be improved when the functions are strongly convex or have Lipschitz continuous gradients?
• RQ3 What is the theoretical convergence rate of the double smoothing method when the effective domain of the functions is unbounded?
• RQ4 How can an approximately optimal primal solution be efficiently recovered from the dual iterates in the double smoothing framework?
• RQ5 How does the accelerated double smoothing method compare in practice to standard first-order methods like ISTA and FISTA in image reconstruction tasks?

Key findings

• When the function $g$ is strongly convex, the convergence rate remains $O\left(\frac{1}{\epsilon}\ln\left(\frac{1}{\epsilon}\right)\right)$ even without requiring bounded effective domain.
• If $f$ is strongly convex or $g$ is everywhere differentiable with a Lipschitz continuous gradient, the convergence rate improves to $O\left(\frac{1}{\sqrt{\epsilon}}\ln\left(\frac{1}{\epsilon}\right)\right)$.
• When both $f$ is strongly convex and $g$ is differentiable with a Lipschitz gradient, the convergence rate achieves the optimal $O\left(\ln\left(\frac{1}{\epsilon}\right)\right)$.
• The double smoothing method successfully reconstructs an approximately optimal primal solution from dual iterates with the same convergence rate as the dual problem.
• Numerical experiments on image deblurring show that the double smoothing method outperforms ISTA and FISTA in terms of signal-to-noise ratio (ISNR) and reconstruction quality.
• The method achieves faster convergence and better image reconstruction quality than standard first-order methods, especially under favorable function properties.