[Paper Review] SGDLibrary: A MATLAB library for stochastic gradient descent algorithms

Hiroyuki Kasai|arXiv (Cornell University)|Oct 27, 2017

Stochastic Gradient Optimization Techniques19 references3 citations

TL;DR

SGDLibrary is a pure-MATLAB, open-source library providing a comprehensive, extensible environment for implementing and evaluating stochastic gradient descent (SGD) algorithms on large-scale machine learning problems. It supports a wide range of optimization methods—including SGD, SVRG, and L-BFGS—enabling researchers to compare convergence, accuracy, and optimality gap across algorithms with minimal code, using built-in visualization and evaluation tools for regression, classification, and matrix completion tasks.

ABSTRACT

We consider the problem of finding the minimizer of a function $f: \mathbb{R}^d ightarrow \mathbb{R}$ of the finite-sum form $\min f(w) = 1/n\sum_{i}^n f_i(w)$. This problem has been studied intensively in recent years in the field of machine learning (ML). One promising approach for large-scale data is to use a stochastic optimization algorithm to solve the problem. SGDLibrary is a readable, flexible and extensible pure-MATLAB library of a collection of stochastic optimization algorithms. The purpose of the library is to provide researchers and implementers a comprehensive evaluation environment for the use of these algorithms on various ML problems.

Motivation & Objective

• To provide a readable, flexible, and extensible MATLAB-based environment for evaluating stochastic optimization algorithms on large-scale machine learning problems.
• To address the lack of a comprehensive, modular, and transparent evaluation framework for SGD and related algorithms in MATLAB.
• To enable researchers to easily compare convergence behavior, classification accuracy, and optimality gap across multiple algorithms on diverse ML problems.
• To support both standard and advanced optimization methods, including variance-reduced and quasi-Newton variants, within a unified software architecture.
• To facilitate reproducible experiments by offering built-in tools for visualization, solution tracking, and performance metrics.

Proposed method

• The library uses a modular architecture separating problem descriptors (e.g., logistic regression, SVM) from optimization solvers (e.g., SGD, SVRG, L-BFGS), enabling plug-and-play comparisons.
• It implements standard and advanced stochastic optimization algorithms, including stochastic gradient descent (SGD), stochastic variance reduced gradient (SVRG), and limited-memory BFGS (L-BFGS).
• The framework supports both full and mini-batch gradient updates, with step-size control and convergence monitoring via user-defined stopping criteria.
• Optimal solutions are computed using a high-precision L-BFGS solver with tight tolerance to enable accurate computation of the optimality gap.
• Built-in functions allow for prediction, accuracy calculation, and visualization of convergence paths and solution trajectories in 2D parameter space.
• The library is fully compatible with GNU Octave and hosted on GitHub, ensuring accessibility and extensibility for the research community.

Experimental results

Research questions

• RQ1 How do different stochastic optimization algorithms (e.g., SGD, SVRG, L-BFGS) compare in terms of convergence speed and final solution quality on large-scale ML problems?
• RQ2 To what extent does the choice of step-size strategy and algorithmic variant affect the performance and stability of stochastic optimization in practice?
• RQ3 Can a pure-MATLAB implementation provide sufficient performance and usability for benchmarking advanced optimization methods without sacrificing readability or extensibility?
• RQ4 How effectively can the library visualize convergence behavior and solution paths, especially in low-dimensional parameter spaces?
• RQ5 How does the library enable fair and reproducible evaluation of optimization algorithms across diverse ML problems like regression, classification, and matrix completion?

Key findings

• The library successfully enables direct comparison of SGD, SVRG, and L-BFGS on various ML problems, demonstrating clear differences in convergence speed and final accuracy.
• For $ε$-regularized logistic regression, SVRG achieved higher classification accuracy (0.9423) than SGD (0.9287) on the test set, indicating superior convergence properties.
• The optimality gap was computed using a high-precision L-BFGS solution, with the gap decreasing to below $10^{-6}$ after 1000 iterations, validating the accuracy of the reference solution.
• Convergence animations were successfully generated for 2D parameter spaces, visually illustrating the trajectory and dynamics of solution updates across algorithms.
• The library’s modular design allowed seamless integration of new solvers and problem types, confirming its extensibility and reusability for future research.
• The framework demonstrated robustness across multiple problem types, including linear regression, softmax classification, and linear SVM, with consistent and reproducible evaluation pipelines.