[Paper Review] The Reversible Residual Network: Backpropagation Without Storing Activations

Aidan N. Gomez, Mengye Ren+2 more|arXiv (Cornell University)|Jul 14, 2017

Advanced Neural Network Applications28 references193 citations

TL;DR

RevNets enable backpropagation without storing layer activations, achieving memory savings with nearly identical accuracy to standard ResNets on CIFAR-10/100 and ImageNet, at a modest training-time overhead.

ABSTRACT

Deep residual networks (ResNets) have significantly pushed forward the state-of-the-art on image classification, increasing in performance as networks grow both deeper and wider. However, memory consumption becomes a bottleneck, as one needs to store the activations in order to calculate gradients using backpropagation. We present the Reversible Residual Network (RevNet), a variant of ResNets where each layer's activations can be reconstructed exactly from the next layer's. Therefore, the activations for most layers need not be stored in memory during backpropagation. We demonstrate the effectiveness of RevNets on CIFAR-10, CIFAR-100, and ImageNet, establishing nearly identical classification accuracy to equally-sized ResNets, even though the activation storage requirements are independent of depth.

Motivation & Objective

• Motivate the memory bottleneck in training deep ResNets due to activation storage requirements.
• Introduce Reversible Residual Networks (RevNets) that reconstruct activations to enable backprop without storing activations.
• Demonstrate that RevNets match ResNet performance on CIFAR-10, CIFAR-100, and ImageNet with reduced memory needs.

Proposed method

• Define reversible blocks by partitioning layer activations into x1 and x2 and using additive coupling with residual functions F and G.
• Show forward mappings y1 = x1 + F(x2), y2 = x2 + G(y1) and reversible reconstructions x2 = y2 - G(y1), x1 = y1 - F(x2).
• Derive a backprop procedure (Algorithm 1) that reconstructs activations online and computes gradients without storing all activations.
• Discuss the computational overhead of reversible backprop (~4N operations versus 3N for standard backprop, with practical overhead near 50% when implemented with automatic differentiation).
• Implement RevNets in TensorFlow and evaluate on CIFAR-10/100 and ImageNet, comparing against matched-depth ResNets.

Experimental results

Research questions

• RQ1 Can reversible blocks reconstruct activations exactly during backprop to avoid storing activations?
• RQ2 What is the memory vs. computation trade-off for RevNets compared to standard ResNets?
• RQ3 Do RevNets achieve comparable accuracy to ResNets on standard benchmarks while providing depth-independent memory savings?
• RQ4 What is the impact of numerical reconstruction error on training dynamics and final accuracy?

Key findings

• RevNets achieve nearly identical accuracy to equally-sized ResNets on CIFAR-10, CIFAR-100, and ImageNet across multiple architectures.
• Activation storage in RevNets is effectively independent of network depth, yielding substantial memory savings.
• Backprop with reconstructed activations introduces modest computational overhead (approximately 33-50% more operations in practice).
• Numerical reconstruction error accumulates during training but does not significantly degrade final performance or training efficiency.
• On ImageNet, RevNet-104 achieves top-1 error 23.10% compared to ResNet-101 at 23.01%.