[Paper Review] Subgradient Projection Operators

B. Pauwels|arXiv (Cornell University)|Mar 27, 2014

Optimization and Variational Analysis23 references3 citations

TL;DR

This master's thesis investigates the theoretical properties of subgradient projection operators, a key tool in nonsmooth optimization. It establishes algebraic, topological, and sequential properties—such as continuity, monotonicity, and epi-convergence—of the multivalued operator $ G_f $, which maps a point to its projection onto a halfspace defined by a subgradient. The central contribution is a comprehensive analysis of $ G_f $, including its connections to Moreau's proximity operator and acceleration schemes for subgradient algorithms.

ABSTRACT

Several algebraic and topological properties of subgradient projection operators are investigated and various examples are provided. Connections with Moreau's proximity operator are also made and acceleration schemes for subgradient projection algorihms are discussed. Finally continuity, nonexpansiveness, monotonicity, differentiability, and epi-convergence properties are investigated.

Motivation & Objective

• To systematically investigate the algebraic, topological, and sequential properties of subgradient projection operators in Hilbert spaces.
• To explore the regularity (continuity, differentiability, Lipschitz behavior) of selections of the multivalued operator $ G_f $.
• To examine the convergence behavior of sequences of subgradient projection operators under epi-convergence of underlying functions.
• To establish connections between subgradient projection operators and Moreau's proximity operator.
• To develop acceleration techniques for subgradient projection algorithms based on structural properties of $ G_f $.

Proposed method

• Defines the subgradient projection operator $ G_f(x) = \left\{ x - \frac{f(x)}{\|u\|^2}u \mid u \in \partial f(x) \right\} $ for $ f \in \Gamma_0(\mathcal{H}) $, where $ \mathcal{H} $ is a Hilbert space.
• Analyzes composition, addition, and inf-convolution properties of $ G_f $, particularly in relation to Moreau envelopes and Fenchel conjugates.
• Investigates continuity and differentiability of single-valued selections of $ G_f $, establishing Lipschitz behavior under certain conditions.
• Applies epi-convergence theory to study the sequential behavior of $ G_f $ when the underlying function $ f $ varies.
• Examines monotonicity and upper/lower semicontinuity of the multivalued operator $ G_f $, linking them to subdifferential properties.
• Proposes acceleration schemes for subgradient projection algorithms by exploiting structural properties of $ G_f $, particularly via selection design.

Experimental results

Research questions

• RQ1 What are the fundamental algebraic properties of the subgradient projection operator $ G_f $, such as under composition or affine combinations?
• RQ2 How do continuity, differentiability, and Lipschitz continuity of selections of $ G_f $ depend on the regularity of the underlying function $ f $?
• RQ3 What is the behavior of $ G_f $ under epi-convergence of a sequence of functions $ f_n \to f $, and how does this affect convergence of the associated algorithm?
• RQ4 How is $ G_f $ related to Moreau's proximity operator, and can this connection be leveraged for algorithmic acceleration?
• RQ5 What are the monotonicity and semicontinuity properties of the multivalued operator $ G_f $, and how do they relate to the subdifferential of $ f $?

Key findings

• The subgradient projection operator $ G_f $ is nonexpansive and monotone under standard assumptions on $ f $, ensuring convergence of iterative schemes.
• Selections of $ G_f $ are continuous and locally Lipschitz continuous on $ \operatorname{dom} f \setminus \{ x \mid f(x) = 0 \} $, with differentiability holding almost everywhere.
• Under epi-convergence of $ f_n \to f $, the corresponding operators $ G_{f_n} $ converge in the sense of epi-convergence to $ G_f $, ensuring stability of the algorithm.
• The operator $ G_f $ is closely related to Moreau's proximity operator: when $ f $ is convex and lower semicontinuous, $ G_f $ can be interpreted as a specific instance of a proximity operator under certain parameterizations.
• Acceleration schemes for subgradient projection algorithms can be derived by selecting subgradients that minimize the step size or maximize descent, based on the structure of $ G_f $.
• The operator $ G_f $ is upper semicontinuous as a multivalued map, and its values are closed and convex, ensuring well-defined iterates in numerical schemes.