[Paper Review] Shake-Shake regularization

Xavier Gastaldi|arXiv (Cornell University)|May 21, 2017

Advanced Neural Network Applications19 references313 citations

TL;DR

Shake-Shake replaces the standard summation of parallel branches in multi-branch networks with a stochastic affine combination during training, improving generalization and achieving state-of-the-art CIFAR results.

ABSTRACT

The method introduced in this paper aims at helping deep learning practitioners faced with an overfit problem. The idea is to replace, in a multi-branch network, the standard summation of parallel branches with a stochastic affine combination. Applied to 3-branch residual networks, shake-shake regularization improves on the best single shot published results on CIFAR-10 and CIFAR-100 by reaching test errors of 2.86% and 15.85%. Experiments on architectures without skip connections or Batch Normalization show encouraging results and open the door to a large set of applications. Code is available at https://github.com/xgastaldi/shake-shake

Motivation & Objective

• Motivate regularization for multi-branch networks beyond BatchNorm and dropout.
• Propose a stochastic affine combination of residual branches during training.
• Evaluate Shake-Shake on CIFAR-10 and CIFAR-100 against state-of-the-art baselines.
• Explore training-time versus inference-time behavior and the role of architectural components (skip connections, BN).

Proposed method

• Replace the residual sum x_{i+1}=x_i+F(x_i,W^{(1)})+F(x_i,W^{(2)}) with x_{i+1}=x_i+α_i F(x_i,W^{(1)})+(1−α_i) F(x_i,W^{(2)}) during training, where α_i ∼ Uniform(0,1).
• During test time, set all α_i to their expected value 0.5.
• Update α_i independently before each forward and backward pass to create stochastic forward/backward flows (gradient augmentation).
• Experiment with 2-branch ResNets (and variants in 3-branch context) and compare forward/backward strategies (Shake, Even, Keep, Batch, Image-level updates).
• Investigate regularization strength via backward-pass coefficient β_i.j and forward-pass α_i.j interactions; analyze alignment and correlation between residual branches.

Experimental results

Research questions

• RQ1 Does stochastic affine blending of residual branches improve generalization on CIFAR-10/100 beyond standard regularization methods?
• RQ2 How do forward vs backward perturbations (Shake vs Keep vs Even) and where (layer, image) the coefficients are applied affect performance?
• RQ3 What is the role of architectural elements (skip connections, BatchNorm) in enabling Shake-Shake regularization?
• RQ4 How does Shake-Shake influence inter-branch correlation and alignment across layers?
• RQ5 What controls the strength and dynamics of the regularization, and how can it be tuned?

Key findings

• Shake-Shake with 2x32d/64d/96d branches achieves 3.55%, 2.98%, and 2.86% CIFAR-10 error respectively (average of 3–5 runs), outperforming many single-shot baselines.
• Image-level coefficient application tends to yield stronger regularization effects than block-level or other schemes.
• Removing skip connections or BatchNorm reveals that Shake-Shake can still regularize, but success depends on architecture and hyperparameters; some configurations diverge without BN or with too-strong coupling.
• Correlation between the outputs of the two residual branches decreases under Shake-Shake, suggesting decorrelation promotes diverse learning between branches.
• Backward-pass coefficient design critically affects learning; unintended configurations (e.g., β_i.j = 1−α_i.j) can drastically harm training, indicating sensitivity to coefficient alignment and timing.
• CIFAR-100 results show Shake-Even-Image reduces error to 15.85% on a ResNeXt-29 2x4x64d variant, indicating cross-dataset regularization benefits.