[Paper Review] Momentum with Variance Reduction for Nonconvex Composition Optimization

Ziyi Chen, Yi Zhou|arXiv (Cornell University)|May 15, 2020

Stochastic Gradient Optimization Techniques30 references4 citations

TL;DR

This paper proposes a novel momentum with variance reduction algorithm for nonconvex composition optimization using SPIDER-based variance reduction. It achieves near-optimal sample complexity in both finite-sum and online settings, reduces proximal mapping evaluations compared to prior Katyusha-style methods, and establishes linear convergence under gradient dominance, significantly accelerating convergence over existing approaches.

ABSTRACT

Composition optimization is widely-applied in nonconvex machine learning. Various advanced stochastic algorithms that adopt momentum and variance reduction techniques have been developed for composition optimization. However, these algorithms do not fully exploit both techniques to accelerate the convergence and are lack of convergence guarantee in nonconvex optimization. This paper complements the existing literature by developing various momentum schemes with SPIDER-based variance reduction for non-convex composition optimization. In particular, our momentum design requires less number of proximal mapping evaluations per-iteration than that required by the existing Katyusha momentum. Furthermore, our algorithm achieves near-optimal sample complexity results in both non-convex finite-sum and online composition optimization and achieves a linear convergence rate under the gradient dominant condition. Numerical experiments demonstrate that our algorithm converges significantly faster than existing algorithms in nonconvex composition optimization.

Motivation & Objective

• Address the lack of convergence guarantees for momentum-based variance reduction in nonconvex composition optimization.
• Reduce the number of proximal mapping evaluations per iteration compared to existing Katyusha momentum schemes.
• Achieve near-optimal sample complexity in nonconvex finite-sum and online composition optimization.
• Provide provable convergence guarantees for online nonconvex composition problems with momentum and variance reduction.
• Accelerate practical convergence while maintaining theoretical optimality in nonconvex settings.

Proposed method

• Proposes a new momentum scheme integrated with SPIDER-based variance reduction for nonconvex composition problems.
• Designs a novel momentum update that requires only one proximal mapping evaluation per iteration, reducing computational overhead.
• Employs a two-level sampling strategy: full gradient at periodic intervals and stochastic gradients otherwise, leveraging SPIDER's low-variance gradient estimates.
• Introduces a hybrid hyperparameter selection rule that balances convergence speed and stability, ensuring theoretical guarantees.
• Uses a recursive momentum update with adaptive step sizes and gradient tracking to reduce variance and accelerate convergence.
• Derives theoretical bounds using telescoping series and Lyapunov analysis to establish convergence under nonconvex and gradient dominant conditions.

Experimental results

Research questions

• RQ1 Can a momentum scheme with SPIDER-based variance reduction be designed for nonconvex composition optimization with provable convergence guarantees?
• RQ2 Can the number of proximal mapping evaluations per iteration be reduced compared to existing Katyusha-style momentum methods?
• RQ3 Does the proposed algorithm achieve near-optimal sample complexity in both finite-sum and online nonconvex composition optimization?
• RQ4 Can the algorithm achieve linear convergence under the gradient dominant condition?
• RQ5 Does the algorithm demonstrate significant practical acceleration over existing state-of-the-art methods?

Key findings

• The proposed algorithm achieves near-optimal sample complexity of O(√max(N,n)ε⁻² + N + n) for finite-sum composition problems.
• For online composition optimization, the sample complexity is O(ε⁻³), matching the theoretical lower bound up to logarithmic factors.
• The algorithm requires only one proximal mapping evaluation per iteration, reducing computational cost compared to two evaluations in prior Katyusha-based methods.
• Under the gradient dominant condition, the algorithm achieves linear convergence rate, demonstrating fast practical convergence.
• Numerical experiments confirm that the algorithm converges significantly faster than existing algorithms in nonconvex composition optimization settings.
• Theoretical analysis establishes convergence in expectation with explicit bounds on the suboptimality gap and gradient norm.