[Paper Review] Stochastic Recursive Variance Reduction for Efficient Smooth Non-Convex Compositional Optimization

Huizhuo Yuan, Xiangru Lian+1 more|arXiv (Cornell University)|Dec 31, 2019

Stochastic Gradient Optimization Techniques34 references4 citations

TL;DR

This paper proposes SARAH-Compositional, a novel stochastic recursive variance reduction algorithm for smooth non-convex compositional optimization. By extending the SARAH framework to compositional objectives, it achieves optimal IFO complexity bounds of $\mathcal{O}((n+m)^{1/2}\varepsilon^{-2})$ in the finite-sum case and $\mathcal{O}(\varepsilon^{-3})$ in the online case, outperforming prior methods in theory and practice across portfolio management, reinforcement learning, and SNE tasks.

ABSTRACT

Stochastic compositional optimization arises in many important machine learning tasks such as value function evaluation in reinforcement learning and portfolio management. The objective function is the composition of two expectations of stochastic functions, and is more challenging to optimize than vanilla stochastic optimization problems. In this paper, we investigate the stochastic compositional optimization in the general smooth non-convex setting. We employ a recently developed idea of extit{Stochastic Recursive Gradient Descent} to design a novel algorithm named SARAH-Compositional, and prove a sharp Incremental First-order Oracle (IFO) complexity upper bound for stochastic compositional optimization: $\mathcal{O}((n+m)^{1/2} \varepsilon^{-2})$ in the finite-sum case and $\mathcal{O}(\varepsilon^{-3})$ in the online case. Such a complexity is known to be the best one among IFO complexity results for non-convex stochastic compositional optimization, and is believed to be optimal. Our experiments validate the theoretical performance of our algorithm.

Motivation & Objective

• To address the challenge of optimizing smooth non-convex compositional objectives, where the objective is the composition of two expectations of stochastic functions.
• To develop a variance-reduced algorithm that achieves faster convergence and better theoretical complexity bounds than existing methods for stochastic compositional optimization.
• To extend the success of recursive gradient methods like SARAH to the more complex compositional setting, enabling efficient optimization in real-world ML applications.
• To validate the theoretical improvements through empirical evaluation on risk-averse portfolio management, value function evaluation in RL, and stochastic neighborhood embedding (SNE).

Proposed method

• The proposed SARAH-Compositional algorithm uses recursive gradient updates to reduce variance in the estimation of the full gradient of the composed objective function $\Phi(x) = f(g(x))$.
• It maintains recursive estimates of the gradients of both the inner function $g(x)$ and the outer function $f(g(x))$, updating them efficiently at each iteration using a control variate mechanism.
• The algorithm operates in a two-level stochastic setting: it samples inner and outer functions $g_w(x)$ and $f_v(y)$ independently, and uses mini-batch approximations to compute gradient estimates.
• It leverages the structure of the compositional objective to compute a recursive update of the full gradient, reducing the number of stochastic gradient evaluations required.
• The method is designed to handle both finite-sum and online settings, with distinct convergence analysis for each case.
• Theoretical analysis establishes that the algorithm achieves the best-known IFO complexity for non-convex compositional optimization.

Experimental results

Research questions

• RQ1 Can recursive variance reduction techniques from standard stochastic optimization be adapted to the more complex setting of compositional objectives?
• RQ2 What is the optimal incremental first-order oracle (IFO) complexity achievable for non-convex stochastic compositional optimization?
• RQ3 Does the proposed SARAH-Compositional algorithm outperform existing methods like SCGD, Acc-SCGD, ASC-PG, and VRSC in terms of convergence speed and stability?
• RQ4 How does the algorithm perform across diverse real-world applications such as portfolio management, reinforcement learning, and dimensionality reduction?

Key findings

• SARAH-Compositional achieves an IFO complexity of $\mathcal{O}((n+m)^{1/2}\varepsilon^{-2})$ in the finite-sum case, which is the best known among existing methods.
• In the online setting, the algorithm attains an IFO complexity of $\mathcal{O}(\varepsilon^{-3})$, matching the best-known bound for this class of problems.
• Numerical experiments on risk-averse portfolio management show that SARAH-Compositional converges faster and with greater stability than rival algorithms.
• In reinforcement learning tasks, SARAH-Compositional demonstrates superior convergence speed and lower gradient norm compared to VRSC, especially under small batch sizes and large inner loop counts.
• On the MNIST dataset for SNE, SARAH-Compositional shows significantly better stability and faster decrease in gradient norm than VRSC, with the gradient norm decreasing steadily within each inner loop.
• The algorithm consistently outperforms baseline methods across all evaluated tasks, confirming the theoretical advantages of recursive variance reduction in compositional optimization.