[Paper Review] Critical initialisation in continuous approximations of binary neural networks

Grigore Stâmâtescu, Federica Gerace+3 more|arXiv (Cornell University)|Feb 1, 2019

Neural Networks and Applications13 references4 citations

TL;DR

This paper proposes a novel Markov chain-based derivation of continuous surrogates for binary neural networks, revealing that critical initialization—where mean weights are initialized near ±1—is essential for training deep networks. Contrary to intuition, standard initialization schemes fail; instead, initializing mean weights at ±1 enables stable signal propagation and avoids vanishing/exploding gradients in both deterministic and stochastic surrogates.

ABSTRACT

The training of stochastic neural network models with binary ($\pm1$) weights and activations via continuous surrogate networks is investigated. We derive new surrogates using a novel derivation based on writing the stochastic neural network as a Markov chain. This derivation also encompasses existing variants of the surrogates presented in the literature. Following this, we theoretically study the surrogates at initialisation. We derive, using mean field theory, a set of scalar equations describing how input signals propagate through the randomly initialised networks. The equations reveal whether so-called critical initialisations exist for each surrogate network, where the network can be trained to arbitrary depth. Moreover, we predict theoretically and confirm numerically, that common weight initialisation schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance. This study shows that, contrary to common intuition, the means of the stochastic binary weights should be initialised close to $\pm 1$, for deeper networks to be trainable.

Motivation & Objective

• To address the lack of principled initialization strategies for continuous surrogates of binary neural networks.
• To derive a unified framework for both deterministic and stochastic surrogates using a Markov chain representation of stochastic binary networks.
• To analyze signal propagation at initialization using mean field theory and self-averaging arguments.
• To determine whether critical initialization exists for different surrogate types and identify optimal initialization schemes.
• To resolve inconsistencies in prior work regarding gradient backpropagation and initialization in binary networks.

Proposed method

• Derive continuous surrogates using a Markov chain formulation of stochastic binary neural networks, enabling a unified treatment of both weight and neuron stochasticity.
• Apply the Gaussian central limit theorem to approximate the distribution of neuron activations at each layer, enabling continuous approximation of discrete binary operations.
• Use mean field theory and self-averaging arguments to derive scalar equations describing signal variance and correlation propagation through randomly initialized networks.
• Analyze criticality by determining whether signal variance remains stable across layers, identifying initialization conditions that preserve correlation to arbitrary depth.
• Compare two surrogate types: deterministic (via analytic integration) and stochastic (via local reparameterization trick), evaluating their critical initialization properties.
• Validate theoretical predictions numerically using MNIST classification experiments with varying network depth, number of Monte Carlo samples, and initialization schemes.

Experimental results

Research questions

• RQ1 Can a unified derivation framework be developed for both deterministic and stochastic surrogates of binary neural networks using a Markov chain representation?
• RQ2 Do critical initializations exist for both deterministic and stochastic surrogates of binary neural networks, and if so, what are the conditions?
• RQ3 How does the choice of initialization for the mean weights of stochastic binary neurons affect signal propagation and training performance in deep networks?
• RQ4 Why do standard initialization schemes—commonly used in continuous networks—fail when applied to the mean values of binary weights in surrogate training?
• RQ5 How does the number of Monte Carlo samples in stochastic surrogates affect the performance of the corresponding binary network during training?

Key findings

• The Markov chain formulation enables a unified derivation of both deterministic and stochastic surrogates, including the first extension of the local reparameterization trick to stochastic binary neurons.
• For networks with stochastic binary weights and neurons, only the deterministic surrogate achieves critical initialization; the stochastic surrogate (LRT) does not.
• For networks with stochastic binary weights and continuous neurons, the LRT surrogate can achieve critical initialization, while no deterministic surrogate exists for this case.
• Critical initialization for both surrogate types requires setting the mean weights close to ±1, a counterintuitive result that contradicts standard initialization heuristics.
• Numerical experiments confirm that initializing mean weights near ±1 leads to significantly better training performance, especially in deeper networks, while standard initialization schemes fail.
• Performance of stochastic binary networks improves with more Monte Carlo samples, approaching that of the continuous surrogate, and converges toward the surrogate's behavior as training progresses.