[Paper Review] What Information Does a ResNet Compress?

Luke Nicholas Darlow, Amos Storkey|arXiv (Cornell University)|Mar 13, 2020

Generative Adversarial Networks and Image Synthesis22 references4 citations

TL;DR

This paper investigates whether the information bottleneck (IB) principle applies to modern ResNet architectures on realistic image data by estimating mutual information (MI) between hidden layers and inputs using PixelCNN++ decoders. It finds two-stage learning—initial MI increase followed by compression—where ResNets discard class-irrelevant details (e.g., background colors, textures), resulting in varied yet semantically faithful samples, visually demonstrating effective compression for generalization.

ABSTRACT

The information bottleneck principle (Shwartz-Ziv & Tishby, 2017) suggests that SGD-based training of deep neural networks results in optimally compressed hidden layers, from an information theoretic perspective. However, this claim was established on toy data. The goal of the work we present here is to test whether the information bottleneck principle is applicable to a realistic setting using a larger and deeper convolutional architecture, a ResNet model. We trained PixelCNN++ models as inverse representation decoders to measure the mutual information between hidden layers of a ResNet and input image data, when trained for (1) classification and (2) autoencoding. We find that two stages of learning happen for both training regimes, and that compression does occur, even for an autoencoder. Sampling images by conditioning on hidden layers' activations offers an intuitive visualisation to understand what a ResNets learns to forget.

Motivation & Objective

• To test whether the information bottleneck (IB) principle holds in realistic deep learning settings with modern ResNet architectures.
• To develop a method for estimating mutual information (MI) between hidden layers and input images in high-dimensional, real-world data.
• To visualize what information ResNets learn to compress during training, especially class-irrelevant features.
• To compare MI dynamics and representation quality in both classification and autoencoding training regimes.
• To provide empirical evidence of compression in deep networks using intuitive, conditional image generation.

Proposed method

• Estimate a lower bound on mutual information (MI) between ResNet hidden layers and input images using conditional PixelCNN++ models as inverse decoders.
• Train PixelCNN++ to generate images conditioned on activations from specific ResNet layers (e.g., h₃, h₄), enabling visual interpretation of preserved information.
• Track MI changes over training time in both classification and autoencoding setups to identify learning stages.
• Use conditional sampling from trained decoders to illustrate how representations evolve—showing what is retained or discarded.
• Compare MI values at initialization, early training, and final weights to detect compression trends.
• Apply the information bottleneck framework to a modern, deep convolutional network (ResNet) on ImageNet-like data, extending prior toy-model studies.

Experimental results

Research questions

• RQ1 Does the information bottleneck principle manifest in a modern ResNet trained on realistic image data?
• RQ2 How does mutual information between hidden layers and input images evolve during training in classification and autoencoding regimes?
• RQ3 What types of information does a ResNet learn to compress—specifically, which features are discarded and which are retained?
• RQ4 Can conditional image generation from hidden layer activations provide an intuitive visualization of representation compression?
• RQ5 Is there a two-stage learning process, with an initial increase in MI followed by a sustained decrease, as predicted by the IB framework?

Key findings

• Two distinct learning stages were observed: an initial, short-lived increase in mutual information (MI) between hidden layers and input data, followed by a prolonged decrease, indicating compression.
• Mutual information between the third ResNet layer (h₃) and input images decreased significantly by 200 training epochs, confirming compression in hidden representations.
• Conditional samples generated from later-stage activations (e.g., 200 epochs) showed greater variation while preserving class identity, demonstrating effective suppression of class-irrelevant features such as background colors and textures.
• Even at initialization, random weights allowed sufficient information to propagate to generate recognizable but poor-quality samples, indicating that initial representations are not fully random or unstructured.
• The network learned foreground and background color partial-invariance: specific colors were discarded, resulting in diverse but semantically consistent samples (e.g., horses with varied sky or grass colors).
• The PixelCNN++ decoder framework successfully visualized the quality of compressed representations, showing that compressed features retain task-relevant structure while discarding noise and irrelevant details.