[Paper Review] Distributed Stochastic Optimization of the Regularized Risk

Shin Matsushima, Hyokun Yun+2 more|arXiv (Cornell University)|Jun 17, 2014

Stochastic Gradient Optimization Techniques44 references6 citations

TL;DR

This paper proposes Distributed Stochastic Optimization (DSO), a novel algorithm that reformulates regularized risk minimization as a saddle-point problem to enable efficient, scalable parallel stochastic optimization. The method achieves near-linear speedup with the number of processors and outperforms existing parallel stochastic and batch solvers in convergence and generalization performance on large-scale datasets.

ABSTRACT

Many machine learning algorithms minimize a regularized risk, and stochastic optimization is widely used for this task. When working with massive data, it is desirable to perform stochastic optimization in parallel. Unfortunately, many existing stochastic optimization algorithms cannot be parallelized efficiently. In this paper we show that one can rewrite the regularized risk minimization problem as an equivalent saddle-point problem, and propose an efficient distributed stochastic optimization (DSO) algorithm. We prove the algorithm's rate of convergence; remarkably, our analysis shows that the algorithm scales almost linearly with the number of processors. We also verify with empirical evaluations that the proposed algorithm is competitive with other parallel, general purpose stochastic and batch optimization algorithms for regularized risk minimization.

Motivation & Objective

• To address the challenge of efficiently parallelizing stochastic optimization for large-scale machine learning.
• To develop a distributed stochastic algorithm that maintains convergence guarantees and generalization performance on massive datasets.
• To overcome the limitations of existing parallel stochastic methods like PSGD, which often converge to suboptimal solutions.
• To provide a unified framework applicable to both smooth and non-smooth regularized risk minimization problems, including SVMs and logistic regression.
• To empirically validate the scalability and effectiveness of the proposed algorithm on tera-scale datasets.

Proposed method

• The paper reformulates the regularized risk minimization problem as a saddle-point problem using duality, enabling distributed optimization via alternating minimization.
• The algorithm uses stochastic gradient updates based on randomly sampled data points, with each processor maintaining local model parameters and dual variables.
• A key innovation is the use of a dual decomposition approach that allows independent updates across processors while preserving convergence to the global optimum.
• The method leverages the equivalence between the original regularized risk problem and the saddle-point formulation, ensuring that the solution remains optimal under distributed computation.
• The algorithm employs a step size schedule that ensures convergence, with theoretical analysis proving near-linear scaling with the number of processors.
• The approach is designed to handle both smooth (e.g., logistic regression) and non-smooth (e.g., SVMs) loss functions and regularizers.

Experimental results

Research questions

• RQ1 Can regularized risk minimization be reformulated as a saddle-point problem to enable efficient distributed stochastic optimization?
• RQ2 Does the proposed DSO algorithm achieve near-linear scalability with the number of processors in distributed settings?
• RQ3 How does DSO compare in convergence speed and generalization performance to state-of-the-art batch and stochastic optimizers like BMRM and SGD?
• RQ4 Can DSO maintain high test accuracy on tera-scale datasets where sub-sampling degrades performance?
• RQ5 Is the convergence of DSO preserved under distributed, asynchronous updates, and can it be further improved with adaptive step sizes?

Key findings

• DSO achieves near-linear speedup with the number of processors, demonstrating strong scalability in distributed environments.
• On the webspam-t dataset, DSO converges to solutions of comparable quality to BMRM and SGD, while outperforming PSGD in final solution quality.
• On the 50M-sample splice site recognition dataset, DSO shows steadily improving AUPRC over time, indicating strong generalization performance even with partial training.
• The algorithm maintains high test accuracy on tera-scale data without sub-sampling, confirming its suitability for massive datasets.
• Empirical results show that DSO's test error decreases steadily with time, unlike batch solvers such as BMRM, which exhibit fluctuating performance during training.
• Theoretical analysis confirms that DSO converges to the global optimum without requiring strong convexity, broadening its applicability.