[Paper Review] New Optimisation Methods for Machine Learning

Aaron Defazio|arXiv (Cornell University)|Oct 9, 2015

Machine Learning and Algorithms2 references3 citations

TL;DR

This thesis introduces Finito and SAGA, two novel optimization algorithms for machine learning problems involving finite sums and graph-structured objectives. Finito achieves fast convergence for strongly convex problems with a proven linear rate, while SAGA offers a general-purpose method with provable convergence for non-strongly convex and non-differentiable regularized problems, both outperforming prior methods in theory and practice.

ABSTRACT

A thesis submitted for the degree of Doctor of Philosophy of The Australian National University. In this work we introduce several new optimisation methods for problems in machine learning. Our algorithms broadly fall into two categories: optimisation of finite sums and of graph structured objectives. The finite sum problem is simply the minimisation of objective functions that are naturally expressed as a summation over a large number of terms, where each term has a similar or identical weight. Such objectives most often appear in machine learning in the empirical risk minimisation framework in the non-online learning setting. The second category, that of graph structured objectives, consists of objectives that result from applying maximum likelihood to Markov random field models. Unlike the finite sum case, all the non-linearity is contained within a partition function term, which does not readily decompose into a summation. For the finite sum problem, we introduce the Finito and SAGA algorithms, as well as variants of each. For graph-structured problems, we take three complementary approaches. We look at learning the parameters for a fixed structure, learning the structure independently, and learning both simultaneously. Specifically, for the combined approach, we introduce a new method for encouraging graph structures with the "scale-free" property. For the structure learning problem, we establish SHORTCUT, a O(n^{2.5}) expected time approximate structure learning method for Gaussian graphical models. For problems where the structure is known but the parameters unknown, we introduce an approximate maximum likelihood learning algorithm that is capable of learning a useful subclass of Gaussian graphical models.

Motivation & Objective

• To develop new optimization methods that efficiently solve large-scale machine learning problems with finite sum structures.
• To address the limitations of existing methods in handling non-strongly convex and non-differentiable regularized problems.
• To design scalable algorithms for learning parameters and structures in Gaussian graphical models.
• To provide a unified theoretical framework with provable convergence for new classes of optimization problems in machine learning.

Proposed method

• Proposes Finito, a dual incremental gradient method that achieves linear convergence for strongly convex finite sum problems using a variance-reduced approach with a midpoint update rule.
• Introduces SAGA, a novel stochastic optimization method with a simple and general theory that supports non-strongly convex and non-differentiable regularizers without additional regularization.
• Develops Prox-Finito as a primal-dual variant of Finito, enabling efficient optimization with non-smooth regularizers through a midpoint-based update scheme.
• Introduces a new method to encourage scale-free graph structures during structure learning in Gaussian graphical models.
• Proposes SHORTCUT, an O(n^2.5) expected-time approximate algorithm for learning Gaussian graphical model structures.
• Develops an approximate maximum likelihood learning algorithm for Gaussian graphical models with known structures, enabling efficient parameter estimation.

Experimental results

Research questions

• RQ1 Can a new optimization method be designed that achieves faster convergence for finite sum problems with strong convexity?
• RQ2 Can a general-purpose optimization method be developed that handles non-strongly convex and non-differentiable regularized problems with provable convergence?
• RQ3 Can effective algorithms be constructed for learning both structure and parameters in Gaussian graphical models?
• RQ4 Can the scale-free property be effectively encouraged in learned graph structures?
• RQ5 Can approximate structure learning be performed in sub-quadratic time for Gaussian graphical models?

Key findings

• Finito achieves a linear convergence rate for strongly convex finite sum problems, with state-of-the-art empirical performance on five benchmark datasets.
• SAGA provides the first provably convergent method for non-strongly convex finite sum problems with non-differentiable regularizers, without requiring additional regularization.
• SAGA demonstrates superior performance on sparse problems compared to Finito, due to its more flexible and robust update mechanism.
• SHORTCUT achieves O(n^2.5) expected time complexity for approximate structure learning in Gaussian graphical models, significantly improving scalability.
• The proposed approximate maximum likelihood algorithm enables efficient parameter learning for a useful subclass of Gaussian graphical models.
• Theoretical analysis confirms convergence for all proposed methods, with tight bounds established for Finito and Prox-Finito.