[Paper Review] Variable Metric Stochastic Approximation Theory

Peter Sunehag, Jochen Trumpf+2 more|ArXiv.org|Aug 25, 2009

Advanced Bandit Algorithms ResearchDecision Sciences15 references17 citations

TL;DR

This paper establishes a convergence theory for variable metric stochastic approximation, proving that online optimization methods using adaptive Hessian approximations—such as online BFGS and LBFGS—converge under general conditions. The key contribution is a rigorous theoretical foundation showing O(1/t) convergence in function values for a broad class of machine learning objectives, even without requiring the metric to stabilize over time.

ABSTRACT

We provide a variable metric stochastic approximation theory. In doing so, we provide a convergence theory for a large class of online variable metric methods including the recently introduced online versions of the BFGS algorithm and its limited-memory LBFGS variant. We also discuss the implications of our results for learning from expert advice.

Motivation & Objective

• To develop a theoretical framework for variable metric stochastic approximation in online learning settings.
• To prove convergence of online quasi-Newton methods like oBFGS and oLBFGS under minimal assumptions.
• To extend existing stochastic approximation theory by removing the need for convergence of the metric to an asymptotic form.
• To analyze implications for learning from expert advice using Bregman divergences.
• To establish conditions under which adaptive metrics ensure optimal convergence rates in online optimization.

Proposed method

• Uses the Robbins-Siegmund stochastic convergence theorem to analyze time-varying metric updates.
• Models online updates as $ w_{t+1} = w_t - a_t B_t \nabla_w C(w_t, z_t) $, where $ B_t $ are positive definite symmetric matrices.
• Applies the theory to online BFGS and LBFGS by verifying that the Hessian approximations satisfy required conditions on eigenvalues and gradients.
• Reinterprets expert advice algorithms as stochastic gradient descent under non-linear reparametrization with variable metrics.
• Introduces a trust region parameter $ \lambda $ to bound the largest eigenvalue of $ B_t $, ensuring numerical stability.
• Establishes convergence by verifying that the Jacobian of the reparametrization function has uniformly bounded eigenvalues, ensuring valid metric scaling.

Experimental results

Research questions

• RQ1 Can variable metric stochastic approximation converge without requiring the metric to converge to a fixed asymptotic form?
• RQ2 What conditions ensure O(1/t) convergence in function values for online quasi-Newton methods?
• RQ3 How do Bregman divergences in expert advice relate to stochastic gradient descent with variable metrics?
• RQ4 Can online BFGS and LBFGS be theoretically justified under general assumptions on gradients and Hessian approximations?
• RQ5 Under what conditions does a non-linear reparametrization lead to optimal convergence rates in online learning?

Key findings

• The paper proves O(1/t) convergence in function values for a broad class of online objectives, matching the best possible rate limited by data arrival.
• Convergence is established without requiring the metric matrices $ B_t $ to converge to a fixed limit, extending prior results.
• Online BFGS and LBFGS are shown to converge under mild assumptions, including bounded gradients and uniformly bounded eigenvalues of the scaling matrices.
• The theory applies to learning from expert advice when using Bregman divergences, by interpreting such updates as SGD with variable metrics.
• A trust region parameter $ \lambda $ is introduced to prevent unbounded condition numbers in $ B_t $, ensuring stability.
• The convergence of o(L)BFGS is guaranteed if the Hessian approximations maintain lower and upper eigenvalue bounds, even if not fully stabilized.