[Paper Review] Optimal Stochastic Strongly Convex Optimization with a Logarithmic Number of Projections

Jianhui Chen, Tianbao Yang+3 more|arXiv (Cornell University)|Apr 19, 2013

Sparse and Compressive Sensing Techniques30 references4 citations

TL;DR

This paper proposes Epro-SGD, an epoch-based stochastic gradient descent method for stochastic strongly convex optimization with complex constraints. By applying SGD within epochs and projecting only once per epoch, it achieves an optimal O(1/T) convergence rate with only O(log T) projections, significantly reducing computational cost while maintaining high accuracy and robustness in practice.

ABSTRACT

We consider stochastic strongly convex optimization with a complex inequality constraint. This complex inequality constraint may lead to computationally expensive projections in algorithmic iterations of the stochastic gradient descent~(SGD) methods. To reduce the computation costs pertaining to the projections, we propose an Epoch-Projection Stochastic Gradient Descent~(Epro-SGD) method. The proposed Epro-SGD method consists of a sequence of epochs; it applies SGD to an augmented objective function at each iteration within the epoch, and then performs a projection at the end of each epoch. Given a strongly convex optimization and for a total number of $T$ iterations, Epro-SGD requires only $\\log(T)$ projections, and meanwhile attains an optimal convergence rate of $O(1/T)$, both in expectation and with a high probability. To exploit the structure of the optimization problem, we propose a proximal variant of Epro-SGD, namely Epro-ORDA, based on the optimal regularized dual averaging method. We apply the proposed methods on real-world applications; the empirical results demonstrate the effectiveness of our methods.

Motivation & Objective

• To address the high computational cost of frequent projections in stochastic gradient descent (SGD) for complex constraints such as positive definite or polyhedral sets.
• To develop an efficient algorithm that maintains optimal convergence rates while minimizing the number of expensive projection operations.
• To design a method that is both theoretically sound and empirically effective for large-scale machine learning problems with complex feasible domains.
• To extend the approach via a proximal variant, Epro-ORDA, to exploit structural properties like sparsity in optimization problems.

Proposed method

• The Epro-SGD method organizes optimization into epochs, applying standard SGD to an augmented objective function within each epoch to avoid intermediate projections.
• At the end of each epoch, a single projection ensures feasibility, reducing the total number of projections to O(log T) over T iterations.
• The augmented objective function incorporates the constraint via a logarithmic barrier or regularization, enabling efficient gradient updates without constraint projection during the epoch.
• The method leverages strong convexity and stochastic gradient oracles with bounded variance to ensure convergence in expectation and with high probability.
• Epro-ORDA extends Epro-SGD by integrating the optimal regularized dual averaging (ORDA) method, enabling efficient proximal mapping for structured problems.
• The algorithm uses adaptive step sizes and epoch-dependent learning rates to balance convergence speed and feasibility.

Experimental results

Research questions

• RQ1 Can we reduce the number of projections in stochastic gradient descent for strongly convex problems with complex constraints without sacrificing convergence rate?
• RQ2 Does an epoch-based projection strategy maintain the optimal O(1/T) convergence rate in expectation and with high probability?
• RQ3 Can a proximal variant of the epoch-projection method exploit problem structure such as sparsity or low-rank constraints to further improve efficiency?
• RQ4 How does the proposed method compare to existing one-projection or full-projection SGD methods in terms of convergence speed and computational cost?

Key findings

• Epro-SGD achieves an optimal convergence rate of O(1/T) in both expectation and with high probability, matching the theoretical lower bound for stochastic strongly convex optimization.
• The method requires only O(log T) projections over T iterations, drastically reducing the computational burden of expensive projection operations such as SVD for positive definite constraints.
• In experiments on constrained Lasso and LMNN, Epro-SGD converged faster than SGD, OneProj, and LogT, achieving lower objective values in fewer iterations and less computation time.
• On the Cora dataset, Epro-SGD reached convergence in 3622 seconds, outperforming OneProj and LogT, which required significantly more time despite similar objective values.
• The Epro-ORDA variant successfully leveraged problem structure, enabling efficient proximal updates and maintaining the logarithmic projection count while preserving optimal convergence.
• Empirical results show that Epro-SGD is not only theoretically optimal but also practically efficient, especially in settings with computationally heavy constraints.