[Paper Review] Estimating or Propagating Gradients Through Stochastic Neurons

Yoshua Bengio|arXiv (Cornell University)|May 14, 2013

Adversarial Robustness in Machine LearningComputer Science15 references81 citations

TL;DR

This paper proposes two novel families of gradient estimators for stochastic neurons in deep learning, enabling back-propagation through non-differentiable, binary stochastic units. The first method uses an unbiased, correlation-based estimator that treats the stochastic neuron's output as a perturbation correlated with the loss, while the second method learns a low-variance correction to a biased estimator, achieving efficient gradient estimation even in non-backpropagatable settings.

ABSTRACT

Stochastic neurons can be useful for a number of reasons in deep learning models, but in many cases they pose a challenging problem: how to estimate the gradient of a loss function with respect to the input of such stochastic neurons, i.e., can we "back-propagate" through these stochastic neurons? We examine this question, existing approaches, and present two novel families of solutions, applicable in different settings. In particular, it is demonstrated that a simple biologically plausible formula gives rise to an an unbiased (but noisy) estimator of the gradient with respect to a binary stochastic neuron firing probability. Unlike other estimators which view the noise as a small perturbation in order to estimate gradients by finite differences, this estimator is unbiased even without assuming that the stochastic perturbation is small. This estimator is also interesting because it can be applied in very general settings which do not allow gradient back-propagation, including the estimation of the gradient with respect to future rewards, as required in reinforcement learning setups. We also propose an approach to approximating this unbiased but high-variance estimator by learning to predict it using a biased estimator. The second approach we propose assumes that an estimator of the gradient can be back-propagated and it provides an unbiased estimator of the gradient, but can only work with non-linearities unlike the hard threshold, but like the rectifier, that are not flat for all of their range. This is similar to traditional sigmoidal units but has the advantage that for many inputs, a hard decision (e.g., a 0 output) can be produced, which would be convenient for conditional computation and achieving sparse representations and sparse gradients.

Motivation & Objective

• To address the challenge of estimating gradients through stochastic neurons, particularly binary stochastic neurons with non-differentiable activation functions.
• To develop methods that allow gradient back-propagation through stochastic units without relying on smooth, differentiable non-linearities.
• To provide an unbiased yet computationally efficient gradient estimator applicable in settings where traditional back-propagation fails, such as in reinforcement learning or models with hard decisions.
• To reduce the high variance of unbiased estimators by learning a correction function that transforms a low-variance but biased estimator into a less biased, lower-variance alternative.
• To connect the proposed estimators to existing frameworks like Boltzmann machines and SPSA, showing theoretical and practical relevance.

Proposed method

• Proposes an unbiased gradient estimator based on the correlation between the stochastic neuron's output and the loss gradient, using the formula $ \mathbb{E}[X_i R] $, where $ X_i $ is the output and $ R $ is the reward.
• Introduces a reward-correlation interpretation of the Boltzmann machine log-likelihood gradient, showing it as an unnormalized form of the correlation-based estimator.
• Develops a variance-reduction technique by training a function to map a biased, low-variance estimator into a less biased, lower-variance estimator with the same variance.
• Applies the method to both binary stochastic neurons and non-linearities that are not flat over their entire range, such as rectifiers.
• Uses a computational graph framework where stochastic units are modeled as $ X_{it} \sim \text{Bernoulli}(\sigma(a_{it})) $, and applies Theorem 1 to derive gradient estimators.
• Demonstrates that the estimator $ X_i^+ - X_i^- $ for biases and $ X_i^+X_j^+ - X_i^-X_j^- $ for weights corresponds exactly to the Boltzmann machine gradient estimator.

Experimental results

Research questions

• RQ1 Can we estimate the gradient of a loss function with respect to the input of a stochastic neuron without assuming small perturbations or smoothness?
• RQ2 Is it possible to construct an unbiased gradient estimator for binary stochastic neurons that does not rely on finite differences or small noise approximations?
• RQ3 How can we reduce the high variance of unbiased gradient estimators in stochastic neural networks while maintaining low computational cost?
• RQ4 Can the Boltzmann machine gradient be interpreted as a form of correlation-based gradient estimator, and what are the implications for training stochastic networks?
• RQ5 What is the relationship between the proposed correlation-based estimator and existing methods like SPSA or reinforcement learning policy gradients?

Key findings

• An unbiased gradient estimator for binary stochastic neurons is derived using the correlation between the neuron's output and the loss, valid even without assuming small perturbations.
• The proposed estimator is computationally cheaper than standard back-propagation because it avoids a backward pass.
• The Boltzmann machine log-likelihood gradient is shown to be equivalent to an unnormalized version of the correlation-based estimator, providing a new interpretation of its learning rule.
• A variance-reduction technique is proposed that learns a correction function to transform a biased, low-variance estimator into a less biased, lower-variance one, preserving variance while reducing bias.
• The method is applicable in settings where traditional back-propagation fails, such as with hard threshold units or in reinforcement learning with future reward estimation.
• Theoretical analysis shows that the correlation-based estimator is fundamentally different from SPSA, as it multiplies the perturbation and reward rather than dividing the change in reward by the perturbation.