[Paper Review] Proximal Averages for Minimization of Entropy Functionals

Heinz H. Bauschke, Scott B. Lindstrom|arXiv (Cornell University)|Jul 24, 2018

Statistical Mechanics and Entropy7 references4 citations

TL;DR

This paper introduces proximal averages as a superior alternative to weighted averages for minimizing entropy functionals, leveraging the Lambert W function to derive closed-form solutions. It demonstrates that proximal averages enable a true homotopy method with flexible barrier enforcement, improving visual and computational fit for entropy minimization under linear constraints.

ABSTRACT

In their 2016 article `Meetings with Lambert $\mathcal{W}$ and Other Special Functions in Optimization and Analysis', Borwein and Lindstrom considered the minimization of an entropy functional utilizing the weighted average of the negative Boltzmann--Shannon entropy $x\mapsto x\log x -x$ and the energy $x \mapsto x^2/2$; the solution employed the Fenchel-Moreau conjugate of the sum. However, in place of a traditional arithmetic average of two convex functions, it is also, perhaps even more, natural to use their proximal average. We explain the advantages and illustrate them by computing the analogous proximal averages for the negative entropy and energy. We use the proximal averages to solve entropy functional minimization problems similar to those considered by Borwein and Lindstrom, illustrating the benefits of using a true homotopy. Through experimentation, we discover computational barriers to obtaining solutions when computing with proximal averages, and we demonstrate a method which appears to remedy them.

Motivation & Objective

• To replace weighted averages with proximal averages in entropy functional minimization for improved homotopy behavior.
• To leverage symbolic computation tools (e.g., Maple's SCAT) to derive closed-form expressions for proximal averages of entropy and energy functions.
• To analyze the computational challenges arising when using proximal averages, particularly near λ = 0 where penalties become extreme.
• To demonstrate that solving the dual problem directly improves convergence speed compared to homotopy-based subproblem iteration.
• To explore the theoretical and computational advantages of proximal averages in convex optimization, especially in relation to epi-continuity and barrier enforcement.

Proposed method

• Use the proximal average construction to interpolate between the negative Boltzmann–Shannon entropy and the quadratic energy function.
• Apply the Fenchel-Moreau conjugate and duality theory to derive closed-form expressions involving the Lambert W function.
• Employ symbolic convex analysis tools (SCAT in Maple) to compute and verify proximal average forms for specific parameters.
• Implement a homotopy method by varying the parameter λ ∈ [0,1] to trace primal solutions continuously.
• Solve the dual problem directly using Newton’s method and gradient descent to improve convergence speed.
• Use the identity $ \mathcal{W}(y)e^{\mathcal{W}(y)} = y $ and its derivatives to simplify and differentiate proximal average expressions.

Experimental results

Research questions

• RQ1 How do proximal averages compare to weighted averages in minimizing entropy functionals, particularly in terms of barrier enforcement and solution flexibility?
• RQ2 What closed-form expressions can be derived for the proximal average of the Boltzmann–Shannon entropy and quadratic energy using the Lambert W function?
• RQ3 What computational challenges arise when using proximal averages, especially as λ → 0, and how can they be mitigated?
• RQ4 Can the dual problem be solved more efficiently than the primal homotopy path, and what are the convergence improvements?
• RQ5 How does the proximal average framework support epi-continuity and continuous transformation of primal solutions in entropy minimization?

Key findings

• The proximal average provides a true homotopy between the entropy and energy functions, enabling flexible barrier enforcement that is not possible with weighted averages.
• For λ = 1, the proximal average solution achieves a constraint violation of 1.66 × 10⁻³ after 2100 gradient descent iterates when solving the dual problem directly.
• When solving the dual problem directly, the error for λ = 1 drops to 1.77 × 10⁻⁶ after 2100 iterates, showing significantly faster convergence than the homotopy method.
• The homotopy method exhibits slower convergence, especially for λ near 0, due to increasingly steep penalties on negative values.
• The use of the Lambert W function enables symbolic derivation of proximal average forms, which are essential for both theoretical analysis and computational implementation.
• The proximal average framework supports epi-continuity, visually demonstrating a continuous transformation of primal solutions across λ ∈ [0,1].