[Paper Review] Scale-invariant Learning by Physics Inversion

Philipp Holl, Vladlen Koltun+1 more|arXiv (Cornell University)|Sep 30, 2021

Machine Learning in Materials Science4 citations

TL;DR

This paper proposes a hybrid training method that replaces the standard gradient from physics simulations with scale-invariant updates from a higher-order inverse solver (e.g., L-BFGS-B), integrating them into a deep learning pipeline. By doing so, it achieves faster convergence and improved solution quality on nonlinear physical inverse problems, outperforming standard first-order optimization even in challenging settings like wave packet localization and exponential function inversion.

ABSTRACT

Solving inverse problems, such as parameter estimation and optimal control, is a vital part of science. Many experiments repeatedly collect data and rely on machine learning algorithms to quickly infer solutions to the associated inverse problems. We find that state-of-the-art training techniques are not well-suited to many problems that involve physical processes. The highly nonlinear behavior, common in physical processes, results in strongly varying gradients that lead first-order optimizers like SGD or Adam to compute suboptimal optimization directions. We propose a novel hybrid training approach that combines higher-order optimization methods with machine learning techniques. We take updates from a scale-invariant inverse problem solver and embed them into the gradient-descent-based learning pipeline, replacing the regular gradient of the physical process. We demonstrate the capabilities of our method on a variety of canonical physical systems, showing that it yields significant improvements on a wide range of optimization and learning problems.

Motivation & Objective

• To address the poor convergence of first-order optimizers like SGD and Adam in highly nonlinear physical systems where gradients vary drastically.
• To overcome the limitations of standard deep learning training, which relies solely on first-order gradients and struggles with ill-conditioned or scale-sensitive problems.
• To integrate higher-order, scale-invariant optimization updates from physics solvers into a standard deep learning framework without requiring Hessian computation over network parameters.
• To improve solution quality and convergence speed in inverse problems involving complex physical processes such as Navier-Stokes flows and wave packet localization.
• To demonstrate that learning-based methods can outperform iterative solvers by leveraging shared parameterization and cross-example gradient updates.

Proposed method

• Replace the adjoint gradient from the physical process with updates computed by a higher-order inverse solver (e.g., L-BFGS-B), which are scale-invariant and better capture local nonlinear behavior.
• Use these physics-derived updates as the effective gradient signal in the backpropagation pipeline, maintaining compatibility with standard deep learning optimizers like Adam.
• Maintain the standard neural network training loop, but substitute the standard gradient of the physical simulation with a physics-informed update vector that approximates the Hessian-informed descent direction.
• Implement the method via a modified loss function: $\tilde{L} = \frac{1}{2}||\mathrm{NN}(y^{*}) - (\mathrm{NN}(\circ) + \Delta x)||_{2}^{2}$, where $\Delta x$ is the physics solver update.
• Apply gradient normalization as a baseline comparison, where gradients are normalized to unit length to mitigate scale issues in exponential function inversion.
• Use standard deep learning components (e.g., ReLU, Sigmoid, fully connected layers) and training procedures, ensuring compatibility with existing frameworks and optimization techniques.

Experimental results

Research questions

• RQ1 Can replacing the standard gradient from a physical simulation with a higher-order, scale-invariant update improve training convergence in nonlinear inverse problems?
• RQ2 How does the proposed hybrid method compare to standard first-order optimization (e.g., Adam, SGD) in terms of solution accuracy and convergence speed on physical systems?
• RQ3 Can a neural network trained with physics-informed updates outperform iterative solvers like L-BFGS-B in localizing wave packets or inverting nonlinear functions such as the exponential map?
• RQ4 To what extent does the method benefit from shared parameterization and cross-example gradient updates in escaping local optima?
• RQ5 Does the approach remain effective and stable when applied to complex physical systems like the Navier-Stokes equations?

Key findings

• The proposed method significantly improves convergence speed and solution quality on inverse problems involving nonlinear physical processes, including wave packet localization and exponential function inversion.
• On the wave packet localization task, the neural network trained with physics-informed updates outperformed L-BFGS-B after 30–40 training iterations, even when L-BFGS-B failed to converge to the global optimum due to poor initial guesses.
• For the exponential function inversion, standard Adam and SGD converged slowly due to gradient scaling issues, but the proposed method with physics-informed updates achieved faster convergence and better accuracy.
• The method successfully mitigated the impact of ill-conditioning and scale variation in gradients, enabling stable and effective training in highly nonlinear settings.
• The integration of higher-order physics updates into the deep learning pipeline maintained compatibility with existing optimization accelerations and stabilizations, enabling practical deployment.
• Empirical results on canonical physical systems, including Navier-Stokes, demonstrated that the hybrid approach yields substantial improvements without requiring Hessian computation over model parameters.