[Paper Review] Stochastic Bound Majorization

Anna Choromanska, Tony Jebara|arXiv (Cornell University)|Sep 22, 2013

Stochastic Gradient Optimization Techniques32 references3 citations

TL;DR

This paper proposes a stochastic bound majorization (SBM) method for optimizing log-linear models that leverages a globally guaranteed quadratic upper bound with second-order curvature information distinct from the Hessian. The method achieves faster convergence and better solution quality than SGD and ASGD in both iteration count and computation time, with a low-rank variant maintaining linear complexity in high-dimensional settings.

ABSTRACT

Recently a majorization method for optimizing partition functions of log-linear models was proposed alongside a novel quadratic variational upper-bound. In the batch setting, it outperformed state-of-the-art first- and second-order optimization methods on various learning tasks. We propose a stochastic version of this bound majorization method as well as a low-rank modification for high-dimensional data-sets. The resulting stochastic second-order method outperforms stochastic gradient descent (across variations and various tunings) both in terms of the number of iterations and computation time till convergence while finding a better quality parameter setting. The proposed method bridges first- and second-order stochastic optimization methods by maintaining a computational complexity that is linear in the data dimension and while exploiting second order information about the pseudo-global curvature of the objective function (as opposed to the local curvature in the Hessian).

Motivation & Objective

• Address the limitations of first-order stochastic methods like SGD, including slow convergence, sensitivity to hyperparameters, and poor performance on regularized problems.
• Overcome the high computational cost of exact second-order methods (e.g., Newton’s method) by developing a stochastic variant that maintains linear complexity in data dimension.
• Bridge the gap between first- and second-order optimization by incorporating global curvature information without computing the full Hessian.
• Develop a low-rank approximation of the curvature bound to scale efficiently to high-dimensional data while preserving second-order benefits.
• Demonstrate empirically that the proposed method converges faster and yields better generalization performance than state-of-the-art stochastic first-order methods.

Proposed method

• Adapt a batch bound majorization method—originally using a quadratic variational upper bound on the log-partition function—to a stochastic setting via iterative data-point updates.
• Apply the Sherman-Morrison formula to the batch update rule to enable incremental, stochastic parameter updates while preserving convergence guarantees.
• Use a curvature matrix derived from a global quadratic bound that approximates the pseudo-global curvature of the objective, differing from the local Hessian.
• Introduce a low-rank approximation of the curvature matrix (with rank $k=1$) to reduce computational cost to $Ó(d)$ per iteration, enabling scalability to high-dimensional data.
• Maintain a fixed step size $η_0 = 1/t$ in the full-rank version, eliminating the need for hyperparameter tuning, and tune $η_0$ only in the low-rank variant.
• Ensure monotonic convergence by construction, as the method is based on majorization, which guarantees non-increasing objective value at each step.

Experimental results

Research questions

• RQ1 Can a stochastic second-order optimization method be developed that maintains linear computational complexity while exploiting global curvature information?
• RQ2 How does the performance of the proposed stochastic bound majorization (SBM) compare to SGD and ASGD in terms of convergence speed and solution quality?
• RQ3 To what extent does the low-rank approximation preserve the benefits of second-order optimization while scaling to high-dimensional data?
• RQ4 Does the method achieve better generalization performance (measured by test likelihood and error) than first-order stochastic methods across diverse datasets?
• RQ5 Can the method be made tuning-free in its full-rank form, and how does this affect convergence behavior?

Key findings

• SBM converges in fewer iterations than SGD and ASGD across all tested datasets, including Mnist, gisette, SecStr, digitl, and Text.
• The low-rank SBM variant achieves the fastest convergence in terms of computation time, outperforming SGD and ASGD in likelihood gain per second.
• SBM achieves higher test log-likelihood and lower test error than both SGD and ASGD, indicating better generalization and solution quality.
• The full-rank SBM method requires no hyperparameter tuning (e.g., step size), as it uses $η_0 = 1/t$ by default, and still outperforms tuned SGD variants.
• Despite its stochastic nature, SBM exhibits monotonic convergence, in contrast to the noisy fluctuations seen in SGD and ASGD.
• The method maintains linear computational complexity ($Ó(d)$) in the data dimension through the low-rank approximation, enabling scalability to high-dimensional problems like Text ($d=23922$).