[Paper Review] Proximal Gradient Algorithms: Applications in Signal Processing

Niccolò Antonello, Lorenzo Stella+2 more|arXiv (Cornell University)|Mar 5, 2018

Sparse and Compressive Sensing Techniques67 references18 citations

TL;DR

This paper presents enhanced proximal gradient (PG) algorithms that combine quasi-Newton methods with forward-adjoint oracles to accelerate convergence in large-scale, nonsmooth, and nonconvex signal processing problems. The framework enables matrix-free optimization, achieving high-accuracy solutions efficiently, and is demonstrated through a high-level modeling language and real-time applications in audio, image, and communication processing.

ABSTRACT

Advances in numerical optimization have supported breakthroughs in several areas of signal processing. This paper focuses on the recent enhanced variants of the proximal gradient numerical optimization algorithm, which combine quasi-Newton methods with forward-adjoint oracles to tackle large-scale problems and reduce the computational burden of many applications. These proximal gradient algorithms are here described in an easy-to-understand way, illustrating how they are able to address a wide variety of problems arising in signal processing. A new high-level modeling language is presented which is used to demonstrate the versatility of the presented algorithms in a series of signal processing application examples such as sparse deconvolution, total variation denoising, audio de-clipping and others.

Motivation & Objective

• Address the limitations of classical optimization methods in solving large-scale, nonsmooth, and nonconvex signal processing problems.
• Improve convergence speed of proximal gradient algorithms without relying on matrix factorizations, enabling real-time and embedded deployment.
• Develop a high-level modeling language to simplify implementation and promote reproducibility in signal processing applications.
• Demonstrate the effectiveness of matrix-free, forward-adjoint oracle-based optimization for problems where traditional methods fail due to scalability.
• Extend the applicability of proximal gradient methods to nonconvex problems, showing their potential for high-performance signal reconstruction.

Proposed method

• Integrate quasi-Newton Hessian approximations into proximal gradient algorithms to accelerate convergence beyond standard first-order methods.
• Utilize forward-adjoint oracles to compute matrix-vector products without storing large matrices, enabling matrix-free optimization.
• Implement a high-level modeling language that allows users to define optimization problems using intuitive syntax, such as `x = Variable(n)` and `@minimize f st cstr`.
• Leverage separable proximal mappings across non-overlapping variable slices to maintain computational efficiency even with multiple constraints.
• Support warm-starting across iterative problems by reusing previous solutions, improving efficiency in sequential optimization tasks.
• Combine the PANOC algorithm with specialized solvers like FPG for tailored performance, and expose algorithm selection via syntax such as `with FPG()`.

Experimental results

Research questions

• RQ1 Can quasi-Newton acceleration significantly improve the convergence speed of proximal gradient methods in large-scale signal processing problems?
• RQ2 To what extent can matrix-free computation via forward-adjoint oracles reduce memory and computational overhead in large-scale optimization?
• RQ3 How effective are proximal gradient algorithms with nonconvex formulations in achieving superior performance compared to convex relaxations?
• RQ4 Can a high-level modeling language simplify the implementation of complex signal processing optimization problems while maintaining efficiency?
• RQ5 What is the practical impact of combining warm-starting and matrix-free operations in iterative signal processing tasks?

Key findings

• The proposed quasi-Newton-enhanced proximal gradient algorithms achieve faster convergence than standard first-order methods, enabling high-accuracy solutions in previously intractable problems.
• Matrix-free computation via forward-adjoint oracles allows efficient handling of large-scale problems without storing or factorizing large matrices.
• The high-level modeling language enables compact, readable, and reproducible code for complex signal processing optimization tasks, such as sparse deconvolution and total variation denoising.
• Nonconvex formulations using proximal gradient methods can yield better application-specific performance and are computationally tractable when spurious local minima are absent.
• The framework supports real-time and embedded deployment due to low memory usage and inherent parallelism in proximal mappings and oracle computations.
• Publicly available code and software tools (e.g., ProxImaL, ODL, ForBES) demonstrate the practicality and reusability of the proposed approach across diverse signal processing domains.