[Paper Review] Analyzing Inverse Problems with Invertible Neural Networks

Lynton Ardizzone, Jakob Kruse+7 more|arXiv (Cornell University)|Aug 14, 2018

Neural Networks and ApplicationsComputer Science154 citations

TL;DR

This paper shows that invertible neural networks (INNs) can estimate the full posterior p(x|y) for ill-posed inverse problems, enabling multi-modal and calibrated parameter inferences in science applications.

ABSTRACT

In many tasks, in particular in natural science, the goal is to determine hidden system parameters from a set of measurements. Often, the forward process from parameter- to measurement-space is a well-defined function, whereas the inverse problem is ambiguous: one measurement may map to multiple different sets of parameters. In this setting, the posterior parameter distribution, conditioned on an input measurement, has to be determined. We argue that a particular class of neural networks is well suited for this task -- so-called Invertible Neural Networks (INNs). Although INNs are not new, they have, so far, received little attention in literature. While classical neural networks attempt to solve the ambiguous inverse problem directly, INNs are able to learn it jointly with the well-defined forward process, using additional latent output variables to capture the information otherwise lost. Given a specific measurement and sampled latent variables, the inverse pass of the INN provides a full distribution over parameter space. We verify experimentally, on artificial data and real-world problems from astrophysics and medicine, that INNs are a powerful analysis tool to find multi-modalities in parameter space, to uncover parameter correlations, and to identify unrecoverable parameters.

Motivation & Objective

• Motivate the need to infer the full posterior p(x|y) for ill-posed inverse problems in natural and life sciences.
• Propose INNs that learn the forward process and implicitly model the inverse via latent variables.
• Show that INNs capture multi-modality, parameter correlations, and unrecoverable parameters.
• Demonstrate competitive performance against ABC and cVAE on synthetic and real-world data.

Proposed method

• Use an invertible neural network f(x)=[y,z] to model the forward process y=s(x) and an implicit inverse g(y,z)=x.
• Introduce latent variables z with prior p(z)=N(0,I) to capture information lost in y, enabling a posterior representation p(x|y).
• Train f and g bi-directionally so that y≈f_y(x) and z≈f_z(x), while enforcing p(z)=N(0,I) and independence of y and z via MMD-based losses.
• Leverage coupling layers and fixed/random permutations to build a fully invertible architecture with tractable Jacobians for exact density computation.
• Apply Maximum Mean Discrepancy (MMD) to enforce distributional matching for z and the joint (y,z) vs. (y,p(z)) and to ensure backward consistency.
• Use forward losses to stabilize learning and an auxiliary loss L_x to improve convergence without changing the optimum.

Experimental results

Research questions

• RQ1 Can INNs recover the full posterior p(x|y) for inverse problems across domains?
• RQ2 How do forward, backward, and bi-directional training strategies affect posterior calibration and multimodality detection?
• RQ3 How do INNs compare to ABC and cVAE in accuracy and posterior calibration on synthetic and real-world data?

Key findings

• INNs can approximate the full posterior p(x|y) and reveal multi-modal and correlated parameter structures.
• Bi-directional training with forward and backward losses yields better posterior calibration than forward-only training.
• INN posteriors show correct unrecoverable parameters and parameter correlations not captured by some baselines.
• On medical data, INN achieves favorable calibration and MAP error vs. cVAE, ABC, and dropout-based NNs.
• In astrophysics applications, INNs recover multimodal posteriors and correlated parameter effects from multispectral data.