[Paper Review] Residual Networks are Exponential Ensembles of Relatively Shallow Networks.

Andreas Veit, Michael J. Wilber+1 more|arXiv (Cornell University)|May 20, 2016

Cell Image Analysis Techniques14 references78 citations

TL;DR

This paper reveals that residual networks (ResNets) function as exponential ensembles of relatively shallow networks, not by preserving gradients across full depth. A large-scale lesion study confirms this ensemble behavior at test time, showing most gradient flow originates from short networks (10–34 layers), implying depth is less critical than implicit multiplicity—the size of the internal ensemble—challenging conventional views on residual learning.

ABSTRACT

In this work, we introduce a novel interpretation of residual networks showing they are exponential ensembles. This observation is supported by a large-scale lesion study that demonstrates they behave just like ensembles at test time. Subsequently, we perform an analysis showing these ensembles mostly consist of networks that are each relatively shallow. For example, contrary to our expectations, most of the gradient in a residual network with 110 layers comes from an ensemble of very short networks, i.e., only 10-34 layers deep. This suggests that in addition to describing neural networks in terms of width and depth, there is a third dimension: multiplicity, the size of the implicit ensemble. Ultimately, residual networks do not resolve the vanishing gradient problem by preserving gradient flow throughout the entire depth of the network - rather, they avoid the problem simply by ensembling many short networks together. This insight reveals that depth is still an open research question and invites the exploration of the related notion of multiplicity.

Motivation & Objective

• To reinterpret residual networks beyond their standard architectural description, focusing on their implicit ensemble nature.
• To investigate whether ResNets behave like ensembles at test time, particularly in terms of gradient contribution.
• To determine the effective depth of individual networks within the ensemble, challenging assumptions about gradient flow through full depth.
• To explore the role of multiplicity—the size of the implicit ensemble—as a new dimension in neural network design, alongside width and depth.

Proposed method

• Conducting a large-scale lesion study by systematically removing residual blocks to measure the impact on test accuracy, simulating ensemble behavior.
• Analyzing gradient flow through the network by measuring the contribution of individual residual blocks to the overall gradient.
• Modeling ResNets as exponential ensembles of shallow subnetworks, where each path corresponds to a different combination of residual blocks.
• Using ablation and gradient attribution to identify which subnetworks contribute most to the final prediction, particularly focusing on short paths.
• Deriving theoretical and empirical support for the claim that the effective model capacity arises from the multiplicative combination of many shallow networks.

Experimental results

Research questions

• RQ1 Do residual networks behave like ensembles at test time, and if so, what is the nature of this ensemble?
• RQ2 What is the effective depth of the individual networks within the ResNet ensemble, and how does it compare to the total network depth?
• RQ3 To what extent does gradient flow in deep ResNets originate from very short subnetworks rather than the full depth?
• RQ4 How does the concept of multiplicity—implicit ensemble size—compare to traditional architectural dimensions like width and depth in determining model performance?

Key findings

• ResNets behave as exponential ensembles of shallow networks at test time, as confirmed by a large-scale lesion study.
• The majority of gradient flow in a 110-layer ResNet originates from subnetworks that are only 10 to 34 layers deep, not from the full depth.
• The effective model capacity is dominated by a large number of short, shallow paths rather than deep, single-path computation.
• ResNets do not resolve the vanishing gradient problem by maintaining gradient flow through the entire depth; instead, they avoid it by implicitly ensembling many short networks.
• The concept of multiplicity—the size of the implicit ensemble—emerges as a critical, previously overlooked dimension in neural network design.