[Paper Review] Resnet in Resnet: Generalizing Residual Architectures

Sasha Targ, Diogo Almeida+1 more|arXiv (Cornell University)|Mar 25, 2016

Advanced Neural Network Applications15 references656 citations

TL;DR

RiR introduces a generalized residual block with parallel residual and transient streams, improving over ResNet and achieving state-of-the-art on CIFAR-100. It generalizes CNNs and ResNets with no extra computational overhead.

ABSTRACT

Residual networks (ResNets) have recently achieved state-of-the-art on challenging computer vision tasks. We introduce Resnet in Resnet (RiR): a deep dual-stream architecture that generalizes ResNets and standard CNNs and is easily implemented with no computational overhead. RiR consistently improves performance over ResNets, outperforms architectures with similar amounts of augmentation on CIFAR-10, and establishes a new state-of-the-art on CIFAR-100.

Motivation & Objective

• Motivate improvements to residual networks by allowing information to be discarded from earlier representations.
• Generalize residual blocks to combine residual and non-residual (transient) streams in parallel.
• Demonstrate that the generalized residual architecture, instantiated as RiR, yields better performance than standard ResNets on CIFAR datasets.
• Show that ResNet Init enables effective implementation of generalized residual blocks with no extra parameters.
• Explore depth robustness and the impact of dual-stream processing on learning residual representations.

Proposed method

• Define a generalized residual block with two parallel streams: a residual stream (r) with identity shortcuts and a transient stream (t) with standard convolutional processing.
• Introduce cross-stream convolutions W_{l,r→t}, W_{l,t→r} and combine r and t through shared activations, batch normalization, and ReLU (Equation 1).
• Implement the generalized block via a modified initialization (ResNet Init) that concatenates streams and allows exact equivalence to separate linear operations (Equation 2).
• Construct RiR by replacing each convolution in a ResNet block with a generalized residual block, enabling variable effective processing depth before addition back to the residual stream.
• Use grid-search-tuned hyperparameters (learning rate schedules, initialization, optimizers, L2) and standard augmentation to compare architectures on CIFAR-10/100.
• Evaluate across various block depths and configurations to assess depth robustness and the contribution of each stream.

Experimental results

Research questions

• RQ1 Does a generalized residual block with dual streams improve expressivity and optimization over standard ResNet blocks?
• RQ2 Can RiR outperform ResNet and ResNet Init on CIFAR-10 and CIFAR-100, including deeper or wider configurations?
• RQ3 How does the distribution of information between residual and transient streams affect learning and final accuracy?
• RQ4 Is ResNet Init a practical and effective way to implement generalized residual blocks without increasing parameters?
• RQ5 How depth and block structure influence RiR performance compared to traditional ResNets?

Key findings

• RiR outperforms the original ResNet in experiments on CIFAR-10 and CIFAR-100.
• ResNet Init consistently improves over standard CNN baselines and ResNet variants.
• RiR achieves state-of-the-art results on CIFAR-100 and competitive performance on CIFAR-10 with standard augmentation.
• Both residual and transient streams contribute to accuracy, and their relative use changes across processing stages.
• RiR enables training deeper residuals more effectively than the standard ResNet, indicating increased depth robustness.