[Paper Review] Benchmarking Invertible Architectures on Inverse Problems

Jakob Kruse, Lynton Ardizzone+2 more|arXiv (Cornell University)|Jan 26, 2021

Model Reduction and Neural NetworksPhysics and Astronomy14 references25 citations

TL;DR

This paper compares ten invertible and related architectures on two low-dimensional inverse problems, finding coupling-layer models (INN, cINN) and simple autoencoders perform best, with MDNs also strong, while some architectures struggle.

ABSTRACT

Recent work demonstrated that flow-based invertible neural networks are promising tools for solving ambiguous inverse problems. Following up on this, we investigate how ten invertible architectures and related models fare on two intuitive, low-dimensional benchmark problems, obtaining the best results with coupling layers and simple autoencoders. We hope that our initial efforts inspire other researchers to evaluate their invertible architectures in the same setting and put forth additional benchmarks, so our evaluation may eventually grow into an official community challenge.

Motivation & Objective

• Motivate systematic evaluation of invertible architectures on inverse problems.
• Assess how hard (invertible) vs soft-invertible models affect reconstruction of p(x|y).
• Identify architectures and losses that yield accurate posterior estimates on benchmark problems.

Proposed method

• Evaluate ten architectures including INN, cINN, IAF, MAF, iResNet, InvAuto, Autoencoder, cVAE, MDN across two inverse problems.
• Use two objective metrics: posterior mismatch (MMD) and forward resimulation error.
• Train with various loss configurations (L2, MMD, ML likelihood) and with optional supervision of forward process.
• Provide ground-truth posteriors via rejection sampling for fair evaluation.
• Include ground-truth posterior sampling for comparison.
• Report inference times and parameter counts to compare efficiency.

Experimental results

Research questions

• RQ1 Is soft invertibility sufficient for solving inverse problems?
• RQ2 Do architectural restrictions for hard invertibility hurt performance?
• RQ3 Which architectures and loss functions yield the most accurate posterior estimates?
• RQ4 How do different models compare in terms of inference speed and scalability?

Key findings

• Coupling-layer based models (INN, cINN) achieve the best average performance across metrics.
• MDN performs very well for both posterior mismatch and resimulation metrics, offering strong direct modeling of p(x|y).
• Invertible ResNet shows mode collapse on the tasks, indicating limitations for these benchmarks.
• IAF and MAF with decoders and some setups fail to capture posteriors robustly in these tasks.
• Some models require full covariance modeling (as with MDN) or careful training to scale to higher dimensions.
• Autoencoders also perform competitively, supporting soft-invertible approaches as viable baselines.