[Paper Review] Online Variance Reduction with Mixtures

Zalán Borsos, Sebastian Curi+2 more|arXiv (Cornell University)|Mar 29, 2019

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper proposes VRM, a novel online variance reduction method that adaptively learns mixture weights over predefined sampling distributions to improve stochastic optimization. By framing the problem as online learning with a restricted simplex projection, VRM achieves sublinear regret of Õ(T⁴/⁵) and empirically outperforms uniform and baseline sampling in deep reinforcement learning, k-means clustering, and hyperparameter selection without prior knowledge.

ABSTRACT

Adaptive importance sampling for stochastic optimization is a promising approach that offers improved convergence through variance reduction. In this work, we propose a new framework for variance reduction that enables the use of mixtures over predefined sampling distributions, which can naturally encode prior knowledge about the data. While these sampling distributions are fixed, the mixture weights are adapted during the optimization process. We propose VRM, a novel and efficient adaptive scheme that asymptotically recovers the best mixture weights in hindsight and can also accommodate sampling distributions over sets of points. We empirically demonstrate the versatility of VRM in a range of applications.

Motivation & Objective

• To address high variance in stochastic optimization due to uniform or suboptimal sampling of data points.
• To leverage prior structural knowledge in data by formulating fixed sampling distributions as mixture components.
• To develop an efficient online learning framework that adaptively tunes mixture weights to minimize variance in gradient estimates.
• To enable effective variance reduction in settings with sets of points (e.g., prioritized experience replay) where standard methods scale poorly.
• To demonstrate the method's versatility and robustness across diverse optimization tasks without requiring prior hyperparameter tuning.

Proposed method

• Formulates adaptive importance sampling with mixtures as an online learning problem over mixture weights.
• Employs the Online Newton Step algorithm with a novel projection onto a restricted simplex to maintain computational efficiency.
• Uses feedback in the form of gradient variance or TD-error to update mixture weights in real time.
• Constructs mixture components as fixed sampling distributions—e.g., based on distance to cluster centers or prioritized experience replay parameters—enabling prior knowledge integration.
• Applies a delayed feedback mechanism in batch settings (e.g., k-means) where gradient norms are observed per batch.
• Employs a two-phase setup: initial hyperparameter tuning on a held-out split to set β and γ, followed by full optimization.

Experimental results

Research questions

• RQ1 Can adaptive mixture weights over fixed sampling distributions significantly reduce variance in stochastic optimization?
• RQ2 Does the proposed VRM algorithm achieve sublinear regret in online variance reduction with mixtures?
• RQ3 Can VRM effectively identify optimal hyperparameters (e.g., ε, α in prioritized replay) in a single training run without prior tuning?
• RQ4 How does VRM compare to uniform sampling and existing adaptive baselines (e.g., VRB) in terms of convergence speed and final performance?
• RQ5 Can VRM scale to sampling distributions over sets of points, such as in experience replay, while maintaining efficiency?

Key findings

• VRM achieves a regret bound of Õ(T⁴/⁵), demonstrating asymptotic convergence to the optimal mixture weights in hindsight.
• In deep reinforcement learning, VRM successfully identifies the best hyperparameter combination (ε, α) in early training, matching or exceeding the performance of the best fixed setting.
• For minibatch k-means, VRM outperforms both uniform sampling and VRB after an initial setup cost, especially on datasets with high c values (e.g., KDD, CSN).
• On MNIST, where the optimal mixture is uniform, VRM performs comparably to uniform sampling, confirming it does not degrade performance when no structure is present.
• The method reduces variance effectively even when sampling distributions are updated over time, such as in dynamic experience replay buffers.
• The framework is computationally efficient and generalizable, handling both point-based and set-based sampling distributions without exponential complexity.