[Paper Review] Relative Performance of Expected and Observed Fisher Information in Covariance Estimation for Maximum Likelihood Estimates

Xumeng Cao|arXiv (Cornell University)|May 5, 2013

Simulation Techniques and Applications49 references4 citations

TL;DR

This paper demonstrates that, under standard regularity conditions for maximum likelihood estimation, the inverse of the expected Fisher information matrix (FIM) outperforms the inverse of the observed FIM in estimating the covariance matrix of the MLE. Using a mean squared error criterion, the study shows asymptotically that the expected FIM yields lower estimation error at the element level, challenging the conventional preference for observed FIM.

ABSTRACT

Maximum likelihood estimation is a popular method in statistical inference. As a way of assessing the accuracy of the maximum likelihood estimate (MLE), the calculation of the covariance matrix of the MLE is of great interest in practice. Standard statistical theory shows that the normalized MLE is asymptotically normally distributed with covariance matrix being the inverse of the Fisher information matrix (FIM) at the unknown parameter. Two commonly used estimates for the covariance of the MLE are the inverse of the observed FIM (the same as the inverse Hessian of the negative log-likelihood) and the inverse of the expected FIM (the same as the inverse FIM). Both of the observed and expected FIM are evaluated at the MLE from the sample data. In this dissertation, we demonstrate that, under reasonable conditions similar to standard MLE conditions, the inverse expected FIM outperforms the inverse observed FIM under a mean squared error criterion. Specifically, in an asymptotic sense, the inverse expected FIM (evaluated at the MLE) has no greater mean squared error with respect to the true covariance matrix than the inverse observed FIM (evaluated at the MLE) at the element level. This result is different from widely accepted results showing preference for the observed FIM. In this dissertation, we present theoretical derivations that lead to the conclusion above. We also present numerical studies on three distinct problems to support the theoretical result. This dissertation also includes two appendices on topics of relevance to stochastic systems. The first appendix discusses optimal perturbation distributions for the simultaneous perturbation stochastic approximation (SPSA) algorithm. The second appendix considers Monte Carlo methods for computing FIMs when closed forms are not attainable.

Motivation & Objective

• To evaluate and compare the performance of expected and observed Fisher information matrices in estimating the covariance of maximum likelihood estimates.
• To assess which estimator—based on expected or observed FIM—yields lower mean squared error in finite-sample and asymptotic settings.
• To challenge the widely held belief that observed FIM is superior for covariance estimation in MLE.
• To provide theoretical justification and numerical validation for the superior performance of the expected FIM under standard MLE regularity conditions.
• To explore practical computation methods for FIMs when closed-form expressions are unavailable.

Proposed method

• Theoretical analysis based on asymptotic distribution theory of MLEs and second-order stochastic expansions.
• Derivation of the mean squared error (MSE) of the inverse observed and inverse expected FIM estimators at the element level.
• Use of asymptotic equivalence between observed and expected FIM under regularity conditions to compare MSE performance.
• Numerical experiments on three distinct statistical models to validate theoretical findings.
• Application of Monte Carlo methods for FIM computation in models without closed-form FIM expressions.
• Incorporation of optimal perturbation distributions for stochastic approximation in SPSA, relevant to FIM estimation.

Experimental results

Research questions

• RQ1 Does the inverse expected Fisher information matrix have lower mean squared error than the inverse observed Fisher information matrix in estimating the MLE covariance matrix?
• RQ2 Under what conditions does the expected FIM outperform the observed FIM in terms of MSE for MLE covariance estimation?
• RQ3 How does the performance of the expected FIM compare to the observed FIM in finite-sample and asymptotic regimes?
• RQ4 Can theoretical MSE comparisons between the two estimators be established under standard MLE regularity conditions?
• RQ5 What are the implications of using expected FIM instead of observed FIM in practical statistical inference?

Key findings

• The inverse expected Fisher information matrix has no greater mean squared error than the inverse observed Fisher information matrix at the element level in the asymptotic regime.
• The superiority of the expected FIM holds under standard regularity conditions for MLE, including smoothness and identifiability of the likelihood function.
• Numerical studies on three distinct models confirm the theoretical prediction that the expected FIM yields lower MSE in covariance estimation.
• The result contradicts the widely accepted view that observed FIM is preferable due to its direct computation from data.
• The expected FIM is shown to be a more accurate estimator of the true MLE covariance matrix under the MSE criterion, even when both are evaluated at the MLE.
• The study provides theoretical and empirical justification for reconsidering the use of expected FIM in standard error estimation for MLE.