[Paper Review] Residual Networks Behave Like Ensembles of Relatively Shallow Networks

Andreas Veit, Michael J. Wilber+1 more|arXiv (Cornell University)|May 20, 2016

Advanced Neural Network Applications17 references600 citations

TL;DR

The paper reframes residual networks as a collection of many paths of varying length, showing they train using short paths and that these paths behave like an ensemble with limited interdependence; long paths contribute little to gradient during training.

ABSTRACT

In this work we propose a novel interpretation of residual networks showing that they can be seen as a collection of many paths of differing length. Moreover, residual networks seem to enable very deep networks by leveraging only the short paths during training. To support this observation, we rewrite residual networks as an explicit collection of paths. Unlike traditional models, paths through residual networks vary in length. Further, a lesion study reveals that these paths show ensemble-like behavior in the sense that they do not strongly depend on each other. Finally, and most surprising, most paths are shorter than one might expect, and only the short paths are needed during training, as longer paths do not contribute any gradient. For example, most of the gradient in a residual network with 110 layers comes from paths that are only 10-34 layers deep. Our results reveal one of the key characteristics that seem to enable the training of very deep networks: Residual networks avoid the vanishing gradient problem by introducing short paths which can carry gradient throughout the extent of very deep networks.

Motivation & Objective

• Propose an unraveled view of residual networks as a collection of multiple paths rather than a single deep chain.
• Investigate whether residual network paths are independent and exhibit ensemble-like behavior.
• Determine which path lengths contribute to gradient flow during training and how this affects training of very deep networks.

Proposed method

• Unravel residual networks to express them as an explicit collection of paths of varying length.
• Conduct lesion studies by deleting or reordering residual blocks to test path independence and ensemble-like behavior.
• Analyze path length distribution, showing a binomial distribution centered near n/2 in an n-block network.
• Measure gradient flow from paths of different lengths to identify the effective training paths (short paths).
• Retrain networks using only the effective paths to test necessity of long paths for performance.

Experimental results

Research questions

• RQ1 Do residual network paths depend strongly on each other or do they exhibit redundancy?
• RQ2 Do the collection of residual paths behave like an ensemble, with smooth performance as paths are added or removed?
• RQ3 What lengths of paths contribute most to gradient flow during training in very deep residual networks?
• RQ4 Are long paths necessary for achieving strong performance, or can training rely on shorter, effective paths?

Key findings

• Deleting single residual modules in residual networks has minimal impact on performance, unlike traditional networks where performance collapses.
• Deleting multiple modules or reordering modules leads to smoothly increasing error, indicating ensemble-like behavior of path collections.
• Most path lengths follow a binomial distribution, with the majority of paths around half the network depth; however, the gradient during training mainly comes from shorter paths (roughly 5–17 blocks in a 54-block network).
• Long paths contribute little to gradient and are not required for training; networks trained with only effective, shorter paths achieve comparable performance to full models.
• Removing downsampling blocks has modest impact, suggesting residual networks rely on multiple parallel paths rather than a single sequential path.