[論文レビュー] i-RevNet: Deep Invertible Networks

Jörn-Henrik Jacobsen, Arnold Smeulders+1 more|arXiv (Cornell University)|Feb 20, 2018

Generative Adversarial Networks and Image Synthesis参考文献 25被引用数 59

ひとこと要約

この論文は i-RevNet を紹介します。完全に可逆な深層ネットワークで、最終分類層まで情報を保持し、非可逆アーキテクチャと同程度の ImageNet 性能を達成しつつ、隠れた表現からの入力を正確に再構成可能にします。

ABSTRACT

It is widely believed that the success of deep convolutional networks is based on progressively discarding uninformative variability about the input with respect to the problem at hand. This is supported empirically by the difficulty of recovering images from their hidden representations, in most commonly used network architectures. In this paper we show via a one-to-one mapping that this loss of information is not a necessary condition to learn representations that generalize well on complicated problems, such as ImageNet. Via a cascade of homeomorphic layers, we build the i-RevNet, a network that can be fully inverted up to the final projection onto the classes, i.e. no information is discarded. Building an invertible architecture is difficult, for one, because the local inversion is ill-conditioned, we overcome this by providing an explicit inverse. An analysis of i-RevNets learned representations suggests an alternative explanation for the success of deep networks by a progressive contraction and linear separation with depth. To shed light on the nature of the model learned by the i-RevNet we reconstruct linear interpolations between natural image representations.

研究の動機と目的

• Motivate whether information loss is necessary for deep representations to generalize to large-scale problems like ImageNet.
• Propose an invertible CNN architecture that avoids discarding information until the final classification layer.
• Demonstrate exact inverse mappings and analyze learned representations for contraction and class separation.
• Compare performance with non-invertible RevNet and ResNet baselines on ImageNet.
• Provide insights into the geometry of representations via reconstruction and interpolation in feature space.

提案手法

• Introduce i-RevNet as a cascade of invertible (homeomorphic) layers that replace non-invertible components of RevNets with invertible ones.
• Use a splitting operator to create two interleaved pathways and invertible downsampling modules S_j that trade spatial resolution for channel width.
• Derive explicit forward and inverse mappings (x_j+1 = S_{j+1} x̃_j; x̃_{j+1} = x_j + F_j x̃_j) and discuss left-inverse and inverse constructions.
• Train two models: an injective i-RevNet (a) and a bijective i-RevNet (b) with comparable layer counts or parameter counts to RevNet/ResNet baselines.
• Evaluate on ImageNet with standard SGD training, compare Top-1 accuracy and parameter counts to ResNet and RevNet baselines.

実験結果

リサーチクエスチョン

• RQ1 Can an invertible CNN preserve all input information up to the final classification while maintaining competitive accuracy on ImageNet?
• RQ2 How does an invertible architecture influence the learned representation in terms of contraction and linear separability with depth?
• RQ3 What does the inverse mapping reveal about the structure of intermediate representations and the feasibility of reconstructing inputs from hidden features?
• RQ4 Do linear projections (e.g., PCA) capture the discriminative subspace effectively in an invertible network's feature space?

主な発見

Architecture	Injective	Bijective	Top-1 error	Parameters
ResNet	-	-	24.7	26M
RevNet	-	-	25.2	28M
i-RevNet (a)	yes	-	24.7	181M
i-RevNet (b)	yes	yes	26.7	29M

• i-RevNets can be fully invertible up to the final classification layer, preserving input information until the last layer.
• Two models were trained: an injective i-RevNet (a) and a bijective i-RevNet (b), achieving competitive results with respective baselines.
• On ImageNet, i-RevNet (a) achieves similar Top-1 performance to RevNet/ResNet with a significantly wider network (181M parameters).
• i-RevNet (b) is bijective with roughly the same parameter count as baselines but shows a 1.5 percentage point drop in Top-1 accuracy compared to the RevNet baseline.
• The inverse Φ^{-1} is numerically stable in reconstruction, with relative inversion errors around 3–5e-6 on ImageNet, despite ill-conditioned local inverses.
• Linear classifiers (e.g., linear SVM) trained on progressively deeper features show improved separability and contraction with depth, indicating a low-dimensional discriminative subspace (e.g., ~200 principal components sufficient for near-full accuracy).