[Paper Review] Variance Reduction in SGD by Distributed Importance Sampling

Guillaume Alain, Alex Lamb+3 more|arXiv (Cornell University)|Nov 20, 2015

Domain Adaptation and Few-Shot Learning13 references87 citations

TL;DR

This paper proposes a distributed stochastic gradient descent (SGD) method that reduces gradient variance by using importance sampling to prioritize the most informative training examples. Workers across multiple machines compute gradient norms in parallel, and a central parameter server selects samples based on these norms to minimize variance, achieving faster convergence even with stale importance weights.

ABSTRACT

Humans are able to accelerate their learning by selecting training materials that are the most informative and at the appropriate level of difficulty. We propose a framework for distributing deep learning in which one set of workers search for the most informative examples in parallel while a single worker updates the model on examples selected by importance sampling. This leads the model to update using an unbiased estimate of the gradient which also has minimum variance when the sampling proposal is proportional to the L2-norm of the gradient. We show experimentally that this method reduces gradient variance even in a context where the cost of synchronization across machines cannot be ignored, and where the factors for importance sampling are not updated instantly across the training set.

Motivation & Objective

• Address the high communication cost and gradient variance in distributed deep learning, particularly in asynchronous SGD settings.
• Reduce reliance on frequent synchronization and bandwidth-heavy gradient transfers by focusing on informative examples.
• Develop a theoretically grounded method to minimize gradient variance using importance sampling with optimal proposal distributions.
• Enable efficient, scalable training by distributing the computation of gradient norms while maintaining convergence guarantees.
• Explore the feasibility of combining this method with existing approaches like asynchronous SGD for improved performance.

Proposed method

• Distribute the computation of per-example gradient norms across multiple workers to identify the most informative training samples.
• Use a centralized parameter server to perform importance sampling based on the L2-norm of individual gradients, selecting samples with higher norms.
• Apply importance sampling with a proposal distribution proportional to the L2-norm of the gradient to minimize variance in the gradient estimate.
• Allow workers to use stale gradient norms due to delayed synchronization, while still maintaining an unbiased and low-variance update.
• Compute and propagate only one floating-point value per example (the importance weight) instead of full gradients, reducing communication overhead.
• Use additive smoothing to stabilize probability weights when sampling, especially in the presence of stale values.

Experimental results

Research questions

• RQ1 Can importance sampling based on gradient norms reduce training variance in distributed SGD when importance weights are stale?
• RQ2 How does the proposed method compare to standard asynchronous SGD in terms of convergence speed and generalization?
• RQ3 To what extent does communication cost reduction via importance sampling improve training efficiency in distributed deep learning?
• RQ4 What is the impact of using stale gradient norms on the performance and stability of the importance sampling scheme?
• RQ5 Can this method be effectively combined with existing distributed training frameworks like asynchronous SGD?

Key findings

• The proposed method significantly reduces gradient variance even when importance weights are computed from stale model parameters.
• Experiments on permutation-invariant SVHN show faster convergence and lower training loss compared to standard SGD, with variance reduction observable even under communication delays.
• The square root of the trace of the covariance matrix (a proxy for gradient variance) is consistently lower under importance sampling, especially when using a smaller learning rate and appropriate smoothing.
• Using a smaller learning rate with additive smoothing (+1.0) yields better variance reduction than a higher learning rate with larger smoothing (+10.0), indicating sensitivity to hyperparameter tuning.
• The method reduces communication cost by transmitting only importance weights (one float per example) instead of full gradients, cutting down network traffic by orders of magnitude.
• The approach remains effective even with delayed updates, suggesting robustness to staleness in distributed settings, though performance depends on proper smoothing of probability weights.