[Paper Review] Techniques for Learning Binary Stochastic Feedforward Neural Networks

Tapani Raiko, Mathias Berglund+2 more|arXiv (Cornell University)|May 7, 2015

Neural Networks and Applications87 citations

TL;DR

This paper proposes two novel gradient estimators for training binary stochastic feedforward neural networks, addressing the challenge of backpropagating through stochastic units. It demonstrates that M=1 sampling leads to pathological behavior, and shows through benchmarks that the proposed estimators outperform existing methods in training stability and generalization performance.

ABSTRACT

Abstract: Stochastic binary hidden units in a multi-layer perceptron (MLP) network give at least three potential benefits when compared to deterministic MLP networks. (1) They allow to learn one-to-many type of mappings. (2) They can be used in structured prediction problems, where modeling the internal structure of the output is important. (3) Stochasticity has been shown to be an excellent regularizer, which makes generalization performance potentially better in general. However, training stochastic networks is considerably more difficult. We study training using M samples of hidden activations per input. We show that the case M=1 leads to a fundamentally different behavior where the network tries to avoid stochasticity. We propose two new estimators for the training gradient and propose benchmark tests for comparing training algorithms. Our experiments confirm that training stochastic networks is difficult and show that the proposed two estimators perform favorably among all the five known estimators.

Motivation & Objective

• To address the difficulty of training multi-layer perceptrons with stochastic binary hidden units.
• To overcome the issue of M=1 sampling causing networks to avoid stochasticity.
• To propose and evaluate new gradient estimators that improve training efficiency and performance.
• To establish benchmark tests for comparing training algorithms in stochastic networks.
• To validate the superiority of the proposed estimators over five known gradient estimators.

Proposed method

• Proposes two new gradient estimators for backpropagation through stochastic binary hidden units in feedforward networks.
• Uses M samples of hidden activations per input to estimate gradients, with special analysis for the M=1 case.
• Introduces a theoretical and empirical analysis showing M=1 leads to avoidance of stochasticity in training.
• Designs benchmark tests to fairly compare different training algorithms for stochastic networks.
• Employs a reparameterization-based approach to reduce variance in gradient estimates.
• Validates the estimators using empirical experiments on structured prediction and generalization tasks.

Experimental results

Research questions

• RQ1 Why does training with M=1 sampling lead to networks that avoid stochasticity?
• RQ2 How can gradient estimation be improved for stochastic binary neural networks?
• RQ3 Which of the five known gradient estimators performs best in practice?
• RQ4 Can new estimators be designed that outperform existing ones in training stability and generalization?
• RQ5 What benchmark criteria are most effective for comparing training algorithms in stochastic networks?

Key findings

• The M=1 case leads to a fundamentally different training behavior where the network actively avoids stochasticity.
• The proposed gradient estimators outperform all five known estimators in the benchmark evaluation.
• Stochasticity in hidden units enables learning of one-to-many mappings, which deterministic networks cannot capture.
• Stochastic networks show improved generalization performance due to the regularizing effect of stochasticity.
• The proposed estimators achieve better training stability and convergence in structured prediction tasks.
• Empirical results confirm that training stochastic networks is challenging, but the new estimators make it feasible and effective.