[Paper Review] Adaptive Importance Sampling for Finite-Sum Optimization and Sampling with Decreasing Step-Sizes

Ayoub El Hanchi, David A. Stephens|arXiv (Cornell University)|Mar 23, 2021

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper proposes Avare, an adaptive importance sampling algorithm for finite-sum optimization and sampling with decreasing step-sizes. By leveraging online learning and variance-reduction techniques, Avare achieves $Ø(T^{2/3})$ and $Ø(T^{5/6})$ dynamic regret for SGD and SGLD respectively, significantly improving convergence in later optimization stages where variance dominates.

ABSTRACT

Reducing the variance of the gradient estimator is known to improve the convergence rate of stochastic gradient-based optimization and sampling algorithms. One way of achieving variance reduction is to design importance sampling strategies. Recently, the problem of designing such schemes was formulated as an online learning problem with bandit feedback, and algorithms with sub-linear static regret were designed. In this work, we build on this framework and propose Avare, a simple and efficient algorithm for adaptive importance sampling for finite-sum optimization and sampling with decreasing step-sizes. Under standard technical conditions, we show that Avare achieves $\mathcal{O}(T^{2/3})$ and $\mathcal{O}(T^{5/6})$ dynamic regret for SGD and SGLD respectively when run with $\mathcal{O}(1/t)$ step sizes. We achieve this dynamic regret bound by leveraging our knowledge of the dynamics defined by the algorithm, and combining ideas from online learning and variance-reduced stochastic optimization. We validate empirically the performance of our algorithm and identify settings in which it leads to significant improvements.

Motivation & Objective

• To address the challenge of variance in stochastic gradient estimators for large-scale finite-sum problems.
• To develop an adaptive importance sampling strategy that minimizes variance dynamically during optimization with decreasing step-sizes.
• To achieve sub-linear dynamic regret bounds for both SGD and SGLD under standard technical conditions.
• To validate empirically the performance gains of the proposed method in practical machine learning settings.
• To extend the framework to mini-batch sampling while preserving unbiasedness and variance reduction.

Proposed method

• Avare formulates importance sampling as an online learning problem with bandit feedback, using partial gradient norm observations.
• It dynamically updates sampling distributions $p^t$ to minimize the trace of the gradient estimator's covariance matrix.
• The algorithm uses a novel mini-batch estimator that combines benefits of sampling without replacement and importance sampling, maintaining unbiasedness.
• It leverages knowledge of the algorithm's dynamics (e.g., decreasing step-sizes) to derive regret bounds.
• The method incorporates a modified epsilon sequence for constant step-size adaptation, ensuring stable performance.
• Regret analysis combines tools from online learning and variance-reduced stochastic optimization to derive dynamic regret bounds.

Experimental results

Research questions

• RQ1 Can adaptive importance sampling achieve sub-linear dynamic regret for SGD and SGLD with decreasing step-sizes?
• RQ2 How does the dynamic regret of adaptive importance sampling compare to static regret in non-stationary settings?
• RQ3 What is the impact of decreasing step-sizes on the effectiveness of variance reduction in stochastic optimization?
• RQ4 Can a mini-batch estimator be designed to preserve unbiasedness while combining sampling without replacement and importance sampling?
• RQ5 How does Avare perform empirically compared to uniform and other importance sampling strategies in real-world datasets?

Key findings

• Avare achieves $Ø(T^{2/3})$ dynamic regret for SGD when using $Ø(1/t)$ step-sizes under standard technical conditions.
• For SGLD, Avare achieves $Ø(T^{5/6})$ dynamic regret under the same step-size regime.
• The algorithm significantly reduces gradient estimator variance, leading to faster convergence in later optimization stages.
• Empirical results on MNIST, IJCNN1, and CIFAR10 show Avare outperforms uniform sampling and other baselines, especially in variance-dominated regimes.
• A modified epsilon sequence enables stable performance with constant step-sizes, suggesting robustness beyond the decreasing step-size assumption.
• The proposed mini-batch estimator maintains unbiasedness while combining the benefits of both sampling without replacement and importance sampling.