[Paper Review] Scalable Stochastic Kriging with Markovian Covariances

Liang Ding, Xiaowei Zhang|arXiv (Cornell University)|Mar 7, 2018

Simulation Techniques and Applications23 references3 citations

TL;DR

This paper introduces Markovian covariance functions (MCFs), a novel class of covariance functions that enable analytically invertible and sparse precision matrices in stochastic kriging. By leveraging a specific functional form tied to linear ODEs, MCFs reduce computational complexity from O(n³) to O(n²), with further gains under low-error or structured design assumptions, enabling scalable, numerically stable large-scale simulation metamodeling.

ABSTRACT

Stochastic kriging is a popular technique for simulation metamodeling due to its exibility and analytical tractability. Its computational bottleneck is the inversion of a covariance matrix, which takes $O(n^3)$ time in general and becomes prohibitive for large n, where n is the number of design points. Moreover, the covariance matrix is often ill-conditioned for large n, and thus the inversion is prone to numerical instability, resulting in erroneous parameter estimation and prediction. These two numerical issues preclude the use of stochastic kriging at a large scale. This paper presents a novel approach to address them. We construct a class of covariance functions, called Markovian covariance functions (MCFs), which have two properties: (i) the associated covariance matrices can be inverted analytically, and (ii) the inverse matrices are sparse. With the use of MCFs, the inversion-related computational time is reduced to $O(n^2)$ in general, and can be further reduced by orders of magnitude with additional assumptions on the simulation errors and design points. The analytical invertibility also enhance the numerical stability dramatically. The key in our approach is that we identify a general functional form of covariance functions that can induce sparsity in the corresponding inverse matrices. We also establish a connection between MCFs and linear ordinary differential equations. Such a connection provides a flexible, principled approach to constructing a wide class of MCFs. Extensive numerical experiments demonstrate that stochastic kriging with MCFs can handle large-scale problems in an both computationally efficient and numerically stable manner.

Motivation & Objective

• To address the computational and numerical instability bottlenecks in stochastic kriging caused by O(n³) covariance matrix inversion for large n.
• To overcome ill-conditioning of large covariance matrices that leads to inaccurate parameter estimation and prediction errors.
• To develop a class of covariance functions that ensure analytical invertibility and sparsity in the precision matrix, enabling efficient and stable computation.
• To establish a principled, flexible framework for constructing such covariance functions using connections to linear ordinary differential equations.

Proposed method

• Propose a new class of covariance functions, termed Markovian covariance functions (MCFs), defined by a specific functional form that induces tridiagonal precision matrices.
• Identify a general functional form where the inverse of the covariance matrix is sparse and analytically computable, reducing inversion complexity from O(n³) to O(n²).
• Leverage the connection between MCFs and linear ODEs to generate a broad, flexible family of valid MCFs with desired properties.
• Use Green’s functions and recurrence relations to derive closed-form expressions for the inverse covariance matrix and its determinant.
• Apply Laplace expansion and induction to prove that the determinant of submatrices formed by removing one or two points follows a recursive structure tied to the MCF parameters.
• Demonstrate that under certain conditions—such as negligible simulation error—the computational cost can be reduced to O(n) by avoiding numerical inversion altogether.

Experimental results

Research questions

• RQ1 Can a class of covariance functions be constructed such that the associated precision matrix is both analytically invertible and sparse, thereby reducing computational complexity in stochastic kriging?
• RQ2 How can the connection between MCFs and linear ordinary differential equations be exploited to generate a flexible and principled family of valid covariance functions?
• RQ3 What is the impact of design point configuration and simulation error structure on the sparsity and invertibility of the precision matrix under MCFs?
• RQ4 To what extent can the O(n³) bottleneck of stochastic kriging be mitigated using MCFs, especially in large-scale simulation settings?
• RQ5 Does the use of MCFs significantly improve numerical stability compared to standard stochastic kriging when n is large?

Key findings

• The proposed Markovian covariance functions (MCFs) ensure that the precision matrix (inverse of the covariance matrix) is analytically invertible and sparse, reducing computational complexity from O(n³) to O(n²).
• For problems with negligible simulation errors, the computational cost is further reduced to O(n), as numerical matrix inversion is no longer required.
• The analytical invertibility of MCF-induced precision matrices dramatically improves numerical stability, mitigating issues from ill-conditioned matrices in large-scale settings.
• The method establishes a direct link between MCFs and solutions of linear ODEs, enabling a systematic and flexible construction of valid covariance functions.
• Numerical experiments confirm that stochastic kriging with MCFs achieves both computational efficiency and numerical stability on large-scale problems.
• The determinant of submatrices formed by removing design points follows a recursive structure tied to the MCF parameters, enabling efficient computation of likelihood and prediction variance.