[Paper Review] Dropout Ensemble Kalman inversion for high dimensional inverse problems

Shiyang Liu, Sebastian Reich+1 more|arXiv (Cornell University)|Aug 31, 2023

Statistical and numerical algorithmsMathematics3 citations

TL;DR

This paper proposes Dropout-Ensemble Kalman Inversion (DEKI), a novel method that overcomes the subspace limitation of standard Ensemble Kalman Inversion (EKI) in high-dimensional inverse problems by applying dropout regularization to ensemble deviation components. DEKI enables convergence with small ensemble sizes, achieves linear computational scaling with dimension, and attains optimal query complexity, as validated by theoretical analysis and numerical experiments on high-dimensional PDE-constrained inverse problems.

ABSTRACT

Ensemble Kalman inversion (EKI) is an ensemble-based method to solve inverse problems. Its gradient-free formulation makes it an attractive tool for problems with involved formulation. However, EKI suffers from the ''subspace property'', i.e., the EKI solutions are confined in the subspace spanned by the initial ensemble. It implies that the ensemble size should be larger than the problem dimension to ensure EKI's convergence to the correct solution. Such scaling of ensemble size is impractical and prevents the use of EKI in high dimensional problems. To address this issue, we propose a novel approach using dropout regularization to mitigate the subspace problem. We prove that dropout-EKI converges in the small ensemble settings, and the computational cost of the algorithm scales linearly with dimension. We also show that dropout-EKI reaches the optimal query complexity, up to a constant factor. Numerical examples demonstrate the effectiveness of our approach.

Motivation & Objective

• Address the subspace property in Ensemble Kalman Inversion (EKI), which restricts solutions to the span of the initial ensemble and necessitates ensemble sizes exceeding the problem dimension.
• Overcome the impracticality of large ensemble sizes in high-dimensional inverse problems by introducing a regularization mechanism that enhances exploration beyond the initial subspace.
• Develop a theoretically grounded, computationally efficient method that maintains convergence and optimality in high-dimensional settings with fixed, small ensemble sizes.
• Establish theoretical convergence guarantees and query complexity bounds for the proposed method, demonstrating its optimality in the context of zeroth-order optimization for inverse problems.

Proposed method

• Introduce dropout regularization to the ensemble deviation components in EKI iterations, randomly suppressing components during updates to promote exploration beyond the initial subspace.
• Modify the standard EKI update rule by applying a Hadamard product between the ensemble deviation and a random dropout mask, effectively introducing stochastic perturbations to the ensemble covariance update.
• Prove exponential convergence for linear and strongly convex nonlinear problems under proper step size adaptation, ensuring stability and accuracy in high-dimensional settings.
• Analyze the query complexity of the algorithm, showing it achieves optimal performance up to a constant factor in the problem class considered, making it efficient for high-dimensional inverse problems.
• Implement the method using a fixed ensemble size regardless of problem dimension, enabling scalability to high-dimensional PDE-constrained inverse problems.
• Use a Tikhonov-regularized loss function with a prior covariance matrix to balance data misfit and prior information, consistent with Bayesian inference frameworks.

Figure 4: Comparison of the inversion solutions of EKI and DEKI in Darcy’s law model in different setups with ensemble size $J=15$ . Row 1: setup 1 $(l_{x}=0.1,l_{y}=0.1,d_{u}=136)$ ; Row 2: setup 2 $(l_{x}=0.2,l_{y}=0.05,d_{u}=142)$ ; Row 3: setup 3 $(l_{x}=0.15,l_{y}=0.05,d_{u}=179)$ ; Row 4: setu

Experimental results

Research questions

• RQ1 Can dropout regularization be effectively adapted to Ensemble Kalman Inversion to mitigate the subspace property and enable convergence in high-dimensional inverse problems with small ensembles?
• RQ2 Does the proposed DEKI method achieve linear computational scaling with respect to problem dimension, making it feasible for large-scale applications?
• RQ3 What is the theoretical convergence rate of DEKI for linear and strongly convex nonlinear problems under appropriate step size selection?
• RQ4 Does DEKI achieve optimal query complexity for zeroth-order optimization in inverse problems, up to a constant factor?
• RQ5 How does DEKI compare to standard EKI in terms of solution accuracy and robustness across high-dimensional, PDE-constrained inverse problems?

Key findings

• DEKI successfully mitigates the subspace property of EKI, enabling convergence to the true solution even with ensemble sizes smaller than the problem dimension.
• Theoretical analysis confirms that DEKI converges exponentially fast for linear problems and in the final stage of strongly convex nonlinear problems when step sizes are properly adapted.
• Computational cost of DEKI scales linearly with the problem dimension $d_u$, which is optimal for high-dimensional inverse problems.
• DEKI achieves optimal query complexity up to a constant factor in the considered problem class, as established through adaptation of zeroth-order optimization complexity analysis.
• Numerical experiments on Darcy’s flow model with up to $d_u = 254$ parameters show that DEKI maintains accurate inversion with $J = 15$ ensemble members, while standard EKI fails to converge.
• In all tested setups, DEKI consistently reduces both data misfit and solution error over time, while EKI stagnates due to subspace confinement, even with repeated random initializations and dropout realizations.