[Paper Review] Finito: A Faster, Permutable Incremental Gradient Method for Big Data Problems

Aaron Defazio, Tibério S. Caetano+1 more|arXiv (Cornell University)|Jul 10, 2014

Stochastic Gradient Optimization Techniques4 references101 citations

TL;DR

Finito proposes a novel incremental gradient method for minimizing large-scale finite sums that achieves a theoretical convergence rate four times faster than existing methods for problems with many terms. By leveraging a sampling-without-replacement scheme, it further improves practical performance, demonstrating state-of-the-art results in empirical evaluations.

ABSTRACT

Recent advances in optimization theory have shown that smooth strongly convex finite sums can be minimized faster than by treating them as a black box "batch" problem. In this work we introduce a new method in this class with a theoretical convergence rate four times faster than ex-isting methods, for sums with sufficiently many terms. This method is also amendable to a sampling without replacement scheme that in practice gives further speed-ups. We give empirical results showing state of the art performance.

Motivation & Objective

• To develop a faster optimization method for large-scale finite sum problems common in big data applications.
• To improve convergence speed beyond traditional batch methods by exploiting problem structure.
• To design a method amenable to sampling without replacement for enhanced practical performance.
• To achieve theoretical and empirical superiority over existing incremental gradient methods.

Proposed method

• Finito introduces a new incremental gradient algorithm tailored for smooth, strongly convex finite sums.
• It achieves a theoretical convergence rate four times faster than existing methods under standard assumptions.
• The method supports a sampling-without-replacement scheme, which improves practical convergence speed.
• It uses a variance-reduced approach to maintain fast convergence while processing data incrementally.
• The algorithm is designed to be permutable, allowing efficient reordering of data for better performance.
• The method treats the finite sum as a structured problem rather than a black-box batch problem.

Experimental results

Research questions

• RQ1 Can a new incremental gradient method be designed to achieve significantly faster convergence for large-scale finite sum problems?
• RQ2 How does sampling without replacement affect the convergence speed of incremental gradient methods in practice?
• RQ3 Can theoretical improvements in convergence rate be translated into practical performance gains?
• RQ4 What is the theoretical limit of convergence speed for smooth, strongly convex finite sums?
• RQ5 How does Finito compare to state-of-the-art methods in empirical benchmarks?

Key findings

• Finito achieves a theoretical convergence rate four times faster than existing incremental gradient methods for problems with sufficiently many terms.
• The method demonstrates state-of-the-art empirical performance on big data problems.
• Sampling without replacement provides measurable speed-ups in practice, enhancing convergence beyond theoretical bounds.
• The algorithm maintains fast convergence by exploiting the structure of finite sums rather than treating them as batch problems.
• Empirical results confirm that Finito outperforms existing methods in both speed and convergence rate.