[Paper Review] A Bregman Extension of quasi-Newton updates II: Convergence and Robustness Properties

Takafumi Kanamori, Atsumi Ohara|arXiv (Cornell University)|Oct 14, 2010

Sparse and Compressive Sensing Techniques15 references3 citations

TL;DR

This paper extends quasi-Newton methods using Bregman divergences as a generalization of the Kullback-Leibler divergence, deriving new Hessian update formulas that generalize DFP and BFGS. It establishes convergence under the new framework and shows that only the standard BFGS formula bounds the influence of inexact line search errors, while other variants can amplify numerical noise, highlighting BFGS's robustness in practice.

ABSTRACT

We propose an extension of quasi-Newton methods, and investigate the convergence and the robustness properties of the proposed update formulae for the approximate Hessian matrix. Fletcher has studied a variational problem which derives the approximate Hessian update formula of the quasi-Newton methods. We point out that the variational problem is identical to optimization of the Kullback-Leibler divergence, which is a discrepancy measure between two probability distributions. Then, we introduce the Bregman divergence as an extension of the Kullback-Leibler divergence, and derive extended quasi-Newton update formulae based on the variational problem with the Bregman divergence. The proposed update formulae belong to a class of self-scaling quasi-Newton methods. We study the convergence property of the proposed quasi-Newton method, and moreover, we apply the tools in the robust statistics to analyze the robustness property of the Hessian update formulae against the numerical rounding errors included in the line search for the step length. As the result, we found that the influence of the inexact line search is bounded only for the standard BFGS formula for the Hessian approximation. Numerical studies are conducted to verify the usefulness of the tools borrowed from robust statistics.

Motivation & Objective

• To extend standard quasi-Newton updates by generalizing the Kullback-Leibler divergence to Bregman divergences for improved Hessian approximation.
• To establish convergence properties of the proposed Bregman-based quasi-Newton update formulas.
• To analyze robustness of Hessian updates against numerical errors from inexact line searches using tools from robust statistics.
• To identify which update formulas maintain bounded influence under perturbations in the line search process.

Proposed method

• Uses Bregman divergence as a generalization of the Kullback-Leibler divergence to define a variational optimization problem for Hessian updates.
• Derives new quasi-Newton update formulas by solving a constrained minimization problem over positive definite matrices using Bregman divergence.
• Applies influence function analysis from robust statistics to assess sensitivity of Hessian updates to perturbations in search direction and gradient information.
• Introduces a parameterized family of update formulas via a function β(z), with the standard BFGS corresponding to β=0.
• Uses matrix perturbation analysis to evaluate the Frobenius norm of the influence function, measuring sensitivity to numerical errors.
• Employs Cholesky factor updates and inverse Hessian updates to maintain computational efficiency in the algorithmic framework.

Experimental results

Research questions

• RQ1 Can the Bregman divergence framework generalize existing quasi-Newton methods like DFP and BFGS?
• RQ2 Do the proposed Bregman-based update formulas maintain convergence under standard quasi-Newton assumptions?
• RQ3 Which update formulas exhibit bounded influence from inexact line search errors in the Hessian approximation?
• RQ4 How does the choice of divergence function (e.g., KL vs. general Bregman) affect the robustness of the Hessian update?
• RQ5 Is the standard BFGS update uniquely robust to numerical noise in the line search compared to other variants?

Key findings

• The proposed Bregman-based update formulas generalize DFP and BFGS and maintain convergence under standard conditions.
• Only the standard BFGS formula bounds the influence of inexact line search errors on the Hessian update, as measured by the influence function's Frobenius norm.
• For all other Bregman-based updates (including DFP and extended variants), the influence function can become unbounded under perturbations, indicating uncontrolled sensitivity to numerical noise.
• The Frobenius norm of the influence function diverges to infinity for all Bregman-based updates except the standard BFGS, even when the determinant of the Hessian approximation is fixed.
• Numerical studies confirm that tools from robust statistics effectively detect and quantify sensitivity to numerical errors in Hessian updates.
• The analysis reveals that BFGS's robustness stems from its symmetric structure and the specific form of its influence function, which remains bounded despite perturbations.