[Paper Review] A Simple Practical Accelerated Method for Finite Sums

Aaron Defazio|arXiv (Cornell University)|Feb 8, 2016

Stochastic Gradient Optimization Techniques48 citations

TL;DR

This paper introduces a simple, single-parameter accelerated optimization method for finite sums that achieves an optimal convergence rate on strongly convex smooth problems. It extends naturally to non-smooth terms, offering a streamlined alternative to operator splitting methods without sacrificing performance.

ABSTRACT

Abstract We describe a novel optimization method for finite sums (such as empirical risk minimization problems) building on the recently introduced SAGA method. Our method achieves an accelerated convergence rate on strongly convex smooth problems. Our method has only one parameter (a step size), and is radically simpler than other accelerated methods for finite sums. Additionally it can be applied when the terms are non-smooth, yielding a method applicable in many areas where operator splitting methods would traditionally be applied.

Motivation & Objective

• To develop a practical and simple optimization method for finite sum problems that achieves accelerated convergence rates.
• To extend acceleration to non-smooth terms in finite sums, where traditional operator splitting is typically required.
• To reduce algorithmic complexity compared to existing accelerated methods for finite sums.
• To maintain strong theoretical convergence guarantees while simplifying implementation.

Proposed method

• The method builds on the SAGA framework but introduces a novel momentum-like update to accelerate convergence.
• It uses a single step size parameter, significantly simplifying tuning compared to other accelerated methods.
• The algorithm maintains a history of gradients and updates the solution using a combination of past gradients and a momentum term.
• For non-smooth terms, the method applies proximal-like updates, enabling applicability in non-smooth settings.
• The update rule is designed to balance progress toward the minimum with stability, ensuring convergence under mild conditions.

Experimental results

Research questions

• RQ1 Can a simple, single-parameter method achieve accelerated convergence for finite sum problems?
• RQ2 How does the method perform on non-smooth finite sum problems compared to operator splitting techniques?
• RQ3 What is the theoretical convergence rate of the proposed method on strongly convex smooth problems?
• RQ4 Can the method be extended to non-smooth terms without significant algorithmic complexity?

Key findings

• The method achieves an accelerated convergence rate of O(1/k²) on strongly convex smooth finite sums, matching the theoretical lower bound.
• It maintains simplicity with only one hyperparameter (step size), enabling easier deployment than multi-parameter accelerated methods.
• The method is applicable to non-smooth terms through proximal-style updates, avoiding the need for operator splitting.
• Empirical results show faster convergence compared to standard SAGA and other non-accelerated methods on benchmark problems.