[Paper Review] Invertible Residual Networks

Jens Behrmann, Will Grathwohl+3 more|arXiv (Cornell University)|Nov 2, 2018

Generative Adversarial Networks and Image Synthesis53 references170 citations

TL;DR

The paper shows standard ResNets can be made invertible (i-ResNets) by enforcing a Lipschitz constraint on each residual block, enabling a single architecture to perform classification and tractable density estimation/generation. Invertibility is achieved with simple training normalization and a fixed-point inverse, yielding competitive discriminative performance and competitive flow-based generative modeling.

ABSTRACT

We show that standard ResNet architectures can be made invertible, allowing the same model to be used for classification, density estimation, and generation. Typically, enforcing invertibility requires partitioning dimensions or restricting network architectures. In contrast, our approach only requires adding a simple normalization step during training, already available in standard frameworks. Invertible ResNets define a generative model which can be trained by maximum likelihood on unlabeled data. To compute likelihoods, we introduce a tractable approximation to the Jacobian log-determinant of a residual block. Our empirical evaluation shows that invertible ResNets perform competitively with both state-of-the-art image classifiers and flow-based generative models, something that has not been previously achieved with a single architecture.

Motivation & Objective

• Bridge discrimination and generative modeling by making ResNets invertible without partitioning dimensions.
• Guarantee invertibility via a Lipschitz constraint on residual blocks.
• Enable likelihood-based training and tractable density estimation for unlabeled data.
• Demonstrate competitive image classification performance and competitive generative modeling results.
• Provide practical guidance for training and using i-ResNets in dual tasks.

Proposed method

• Treat ResNet blocks as Lipschitz-constrained residual mappings Fθ = I + gθt with Lip(gθt) < 1 for all t.
• Invert the layer via a fixed-point iteration to obtain Fθ^{-1} using the Banach fixed-point theorem.
• Enforce the Lipschitz constraint by spectral normalization of each linear layer to ensure ||Wi||2 < 1 (with a conservative scaling c < 1).
• Compute the log-determinant needed for likelihoods using a tractable power-series approximation of tr(log(I + Jg(x))) via Hutchinson’s trace estimator and a truncated series (k terms).
• Provide a forward pass algorithm (Algorithm 2) that combines spectral normalization, stochastic log-determinant estimation, and fixed-point inverse computation.
• Discuss the relation to Neural ODEs and other reversible architectures, and compare analytic/inverse properties across models.
• Train i-ResNets as normalizing flows for density estimation; samples generated by sampling z ~ pz and applying the inverse F^{-1}.
• Address scalability to high dimensions by using the log-determinant power-series approximation which scales with the number of terms and layers.

Experimental results

Research questions

• RQ1 Can standard ResNet architectures be made invertible without partitioning dimensions or restrictive design choices?
• RQ2 Does enforcing a Lipschitz constraint enable stable, tractable density estimation and generation while preserving competitive discriminative performance?
• RQ3 How does i-ResNet compare to other invertible architectures (NICE, i-RevNet, Real-NVP, Glow, FFJORD) in classification and generative tasks?
• RQ4 What practical guidelines (e.g., spectral normalization coefficient, number of power-series terms) yield reliable inverse computation and accurate log-determinant estimates?

Key findings

• i-ResNets can be made invertible by enforcing Lip(g) < 1 via spectral normalization without significantly harming classification accuracy on MNIST, CIFAR-10, and CIFAR-100.
• The inverse of an i-ResNet block can be computed with a fixed-point iteration, leveraging Banach’s fixed-point theorem for stable recovery.
• i-ResNets deliver competitive discriminative performance compared to standard ResNets and outperform several Glow variants on CIFAR-10 classification tasks.
• As generative models, i-ResNets show competitive density estimation and sample quality relative to Glow and FFJORD, with performance gaps attributed partly to a biased log-determinant estimator and to optimization challenges.
• The proposed log-determinant approximation (via a truncated power series and Hutchinson trace estimator) scales to higher dimensions and provides tractable likelihood computations for i-ResNet flows.
• Table comparisons indicate i-ResNets achieve competitive bits-per-dimension scores on MNIST and CIFAR-10 (e.g., MNIST around 1.06 bits/dim for i-ResNet versus 0.99 for FFJORD, CIFAR-10 around 3.45 bits/dim).