[Paper Review] Inverting Convolutional Networks with Convolutional Networks.

Alexey Dosovitskiy, Thomas Brox|arXiv (Cornell University)|Jun 9, 2015

Generative Adversarial Networks and Image SynthesisComputer Science20 references77 citations

TL;DR

This paper proposes inverting convolutional neural network (CNN) features using a transposed (up-convolutional) network to reconstruct images from feature representations. It demonstrates that both shallow features (HOG, SIFT, LBP) and deep features from ImageNet-trained networks can be reconstructed with high fidelity, revealing rich structural and color information even from high-level activations and class probabilities.

ABSTRACT

Feature representations, both hand-designed and learned ones, are often hard to analyze and interpret, even when they are extracted from visual data. We propose a new approach to study image representations by inverting them with an up-convolutional neural network. We apply the method to shallow representations (HOG, SIFT, LBP), as well as to deep networks. For shallow representations our approach provides significantly better reconstructions than existing methods, revealing that there is surprisingly rich information contained in these features. Inverting a deep network trained on ImageNet provides several insights into the properties of the feature representation learned by the network. Most strikingly, the colors and the rough contours of an image can be reconstructed from activations in higher network layers and even from the predicted class probabilities.

Motivation & Objective

• To develop a method for reconstructing images from learned or handcrafted visual features to improve interpretability.
• To investigate how much structural and semantic information is preserved in feature representations from shallow and deep networks.
• To analyze the representational capacity of deep features by inverting them back to pixel space.
• To evaluate whether class probabilities alone contain sufficient information for image reconstruction.
• To compare the reconstruction quality of different feature types using a unified inversion framework.

Proposed method

• The method uses a transposed convolutional neural network (deconvnet) to invert feature maps back to pixel space.
• The deconvnet is trained end-to-end to minimize the reconstruction loss between the original input image and the reconstructed image.
• The architecture mirrors the encoder network (e.g., VGG or AlexNet) in a symmetric, deconvolutional fashion to reconstruct spatial details.
• The method is applied to both shallow features (HOG, SIFT, LBP) and deep features from pre-trained ImageNet networks.
• For deep features, reconstruction is performed from activations in higher layers and even from final class probabilities.
• The training process uses a pixel-wise L2 loss to optimize the reconstruction quality.

Experimental results

Research questions

• RQ1 Can a transposed CNN effectively reconstruct images from shallow visual features like HOG, SIFT, and LBP?
• RQ2 To what extent do high-level features in deep networks preserve spatial and color information?
• RQ3 Can image reconstruction be achieved from only the predicted class probabilities of a deep network?
• RQ4 How does the reconstruction quality vary across different layers of a deep network?
• RQ5 What insights into feature representation can be gained by visualizing the inverse process of feature extraction?

Key findings

• The proposed method achieves significantly better reconstruction quality for shallow features (HOG, SIFT, LBP) compared to existing inversion techniques.
• Even from high-level features in deeper layers, the method reconstructs rough contours and color structures of the original image.
• Remarkably, the network can reconstruct coarse color and contour information from only the final class probabilities of an ImageNet classifier.
• The reconstruction from deep features reveals that the network learns hierarchical, semantically meaningful representations with strong spatial coherence.
• The results demonstrate that deep features retain substantial image structure, challenging assumptions about information loss in high-level representations.
• The inversion process provides visual interpretability of feature spaces, enabling analysis of what features actually encode.