[Paper Review] Benchmarking deep inverse models over time, and the neural-adjoint method

Simiao Ren, Willie J. Padilla+1 more|arXiv (Cornell University)|Sep 27, 2020

Neural Networks and Applications27 references20 citations

TL;DR

This paper proposes a time-dependent benchmarking framework for deep inverse models, redefining performance evaluation as a function of inference time (T) rather than a single estimate. It introduces the Neural-Adjoint (NA) method—using a neural network surrogate of the forward model and gradient-based optimization from random initializations—and shows that adding a novel boundary loss significantly improves performance, achieving state-of-the-art results across four benchmark tasks including meta-material design.

ABSTRACT

We consider the task of solving generic inverse problems, where one wishes to determine the hidden parameters of a natural system that will give rise to a particular set of measurements. Recently many new approaches based upon deep learning have arisen generating impressive results. We conceptualize these models as different schemes for efficiently, but randomly, exploring the space of possible inverse solutions. As a result, the accuracy of each approach should be evaluated as a function of time rather than a single estimated solution, as is often done now. Using this metric, we compare several state-of-the-art inverse modeling approaches on four benchmark tasks: two existing tasks, one simple task for visualization and one new task from metamaterial design. Finally, inspired by our conception of the inverse problem, we explore a solution that uses a deep learning model to approximate the forward model, and then uses backpropagation to search for good inverse solutions. This approach, termed the neural-adjoint, achieves the best performance in many scenarios.

Motivation & Objective

• To re-evaluate deep inverse models not by single-estimate error but by performance over time (T), reflecting practical inference efficiency.
• To identify limitations in existing inverse models, such as incomplete exploration or poor localization in x-space.
• To propose and evaluate the Neural-Adjoint (NA) method as a simple, efficient alternative for solving generic inverse problems.
• To demonstrate the effectiveness of a novel boundary loss term in enhancing NA performance.
• To provide a comprehensive benchmark across four diverse inverse problem tasks, including a new 2D sinusoid and meta-material design.

Proposed method

• The paper introduces a new metric, $ r_T $, which computes the expected minimum re-simulation error over T random samples of latent noise z, capturing performance as a function of inference time.
• The Neural-Adjoint (NA) method trains a deep neural network to approximate the forward model f, then uses backpropagation to optimize x by descending the gradient of f(\hat{x}) with respect to x.
• The NA method initializes x from random points in x-space and uses gradient descent to find locally optimal solutions, treating the network as a differentiable simulator.
• A novel boundary loss is introduced to regularize the optimization process, encouraging solutions to stay within feasible or physically meaningful regions of x-space.
• The method is evaluated on four benchmark tasks: ballistics, robotic arm, 2D sinusoid, and meta-material design, using a fast proxy simulator for forward evaluations.
• All models are trained under a fixed 1-day GPU budget, and performance is compared across T values using the $ r_T $ metric.

Experimental results

Research questions

• RQ1 How does the performance of modern inverse models vary with the number of inference samples (T), and does this challenge the validity of single-estimate metrics?
• RQ2 What are the primary failure modes of existing inverse models—poor exploration of x-space or imprecise localization of optimal solutions?
• RQ3 Can a simple differentiable optimization approach using a neural network surrogate for the forward model outperform complex deep generative models?
• RQ4 How effective is the proposed boundary loss in improving the convergence and accuracy of the Neural-Adjoint method?
• RQ5 Does the NA method with boundary loss achieve state-of-the-art performance across diverse inverse problem benchmarks?

Key findings

• The NA method with the boundary loss achieves the best or nearly best performance across all four benchmark tasks, including the challenging meta-material design task.
• On the meta-material design task, the NA method achieved a re-simulation error of $ 2.50 \times 10^{-4} $ at T=1, outperforming cVAE ($ 8.60 \times 10^{-3} $) and INN ($ 3.89 \times 10^{-2} $).
• The NA method required only 0.5–3.1 million parameters (vs. up to 19 million for cVAE), resulting in significantly smaller model size and faster training (e.g., 86s vs. 987s on D1:Ballistics).
• The boundary loss reduced the average re-simulation error of NA by up to an order of magnitude in some tasks, such as the 2D sinusoid task, where error dropped from $ 1.18 \times 10^{-3} $ to $ 7.56 \times 10^{-1} $ without it.
• The $ r_T $ metric revealed that model performance is highly dependent on T, with some models (e.g., cVAE) performing poorly at low T but improving with more samples, highlighting limitations in current evaluation practices.
• The proxy simulator used for fast forward evaluations was released alongside the code, enabling reproducibility and future benchmarking.