[Paper Review] Skip Connections Eliminate Singularities

A. Emin Orhan, Xaq Pitkow|arXiv (Cornell University)|Jan 31, 2017

Stochastic Gradient Optimization Techniques86 citations

TL;DR

The paper argues that skip connections improve training of deep networks by eliminating non-identifiability singularities in the loss landscape (elimination, overlap, and linear dependence), supported by theoretical analysis and empirical results across deep networks and datasets.

ABSTRACT

Skip connections made the training of very deep networks possible and have become an indispensable component in a variety of neural architectures. A completely satisfactory explanation for their success remains elusive. Here, we present a novel explanation for the benefits of skip connections in training very deep networks. The difficulty of training deep networks is partly due to the singularities caused by the non-identifiability of the model. Several such singularities have been identified in previous works: (i) overlap singularities caused by the permutation symmetry of nodes in a given layer, (ii) elimination singularities corresponding to the elimination, i.e. consistent deactivation, of nodes, (iii) singularities generated by the linear dependence of the nodes. These singularities cause degenerate manifolds in the loss landscape that slow down learning. We argue that skip connections eliminate these singularities by breaking the permutation symmetry of nodes, by reducing the possibility of node elimination and by making the nodes less linearly dependent. Moreover, for typical initializations, skip connections move the network away from the "ghosts" of these singularities and sculpt the landscape around them to alleviate the learning slow-down. These hypotheses are supported by evidence from simplified models, as well as from experiments with deep networks trained on real-world datasets.

Motivation & Objective

• Motivate and explain why training very deep networks benefits from skip connections.
• Identify and characterize three types of singularities (elimination, overlap, linear dependence) that slow learning.
• Demonstrate that skip connections reduce degeneracy and accelerate training across architectures and datasets.
• Provide practical alternatives and architectural insights that further mitigate singularities beyond standard residuals.

Proposed method

• Model analysis of three singularities in fully-connected layers: elimination, overlap, and linear dependence.
• Theoretical discussion of how skip connections disrupt singular manifolds and restore identifiability.
• Empirical comparison of plain, residual, and hyper-residual architectures on CIFAR-100/CIFAR-10/100 with Hessian eigenvalue density estimation.
• Introduction of BiasReg as a simple bias-target regularization to break permutation symmetry and elimination singularities.
• Evaluation of non-identity skip schemes, including random dense orthogonal skips, to test symmetry-breaking effects.
• Investigation of gradient norms and vanishing gradients, including the impact of batch normalization on BiasReg networks.

Experimental results

Research questions

• RQ1 Do skip connections remove non-identifiability singularities in deep networks?
• RQ2 How do elimination, overlap, and linear dependence singularities affect learning dynamics and optimization landscapes?
• RQ3 Do skip connections improve training speed and robustness beyond favorable initialization?
• RQ4 Can alternative symmetry-breaking methods (e.g., BiasReg, orthogonal skips) replicate the benefits of skip connections?

Key findings

• Skip connections reduce degeneracy in the Hessian spectrum, correlating with faster training compared to plain nets.
• Hyper-residual architectures exhibit the lowest degeneracy and highest early training speed among studied architectures.
• Bias regularization that targets biases to break symmetry improves performance over plain nets, though not fully matching residuals.
• Orthogonal (dense) skip connections slightly outperform identity skips by better disambiguating units and reducing elimination/overlap risks.
• Malicious initialization results indicate that benefits of skip connections extend beyond initialization, pointing to landscape reshaping near singularities.
• Evidence from both shallow and deep networks suggests singularities are a meaningful bottleneck in optimization that skip connections alleviate.