[论文解读] Relative Performance of Expected and Observed Fisher Information in Covariance Estimation for Maximum Likelihood Estimates

Xumeng Cao|arXiv (Cornell University)|May 5, 2013

Simulation Techniques and Applications参考文献 49被引用 4

一句话总结

本文在最大似然估计的标准正则性条件下表明，期望费雪信息矩阵（FIM）的逆矩阵在估计MLE的协方差矩阵方面优于观测费雪信息矩阵（FIM）的逆矩阵。基于均方误差准则，研究显示在渐近情况下，期望FIM在元素层面上的估计误差更低，挑战了传统上对观测FIM的偏好。

ABSTRACT

Maximum likelihood estimation is a popular method in statistical inference. As a way of assessing the accuracy of the maximum likelihood estimate (MLE), the calculation of the covariance matrix of the MLE is of great interest in practice. Standard statistical theory shows that the normalized MLE is asymptotically normally distributed with covariance matrix being the inverse of the Fisher information matrix (FIM) at the unknown parameter. Two commonly used estimates for the covariance of the MLE are the inverse of the observed FIM (the same as the inverse Hessian of the negative log-likelihood) and the inverse of the expected FIM (the same as the inverse FIM). Both of the observed and expected FIM are evaluated at the MLE from the sample data. In this dissertation, we demonstrate that, under reasonable conditions similar to standard MLE conditions, the inverse expected FIM outperforms the inverse observed FIM under a mean squared error criterion. Specifically, in an asymptotic sense, the inverse expected FIM (evaluated at the MLE) has no greater mean squared error with respect to the true covariance matrix than the inverse observed FIM (evaluated at the MLE) at the element level. This result is different from widely accepted results showing preference for the observed FIM. In this dissertation, we present theoretical derivations that lead to the conclusion above. We also present numerical studies on three distinct problems to support the theoretical result. This dissertation also includes two appendices on topics of relevance to stochastic systems. The first appendix discusses optimal perturbation distributions for the simultaneous perturbation stochastic approximation (SPSA) algorithm. The second appendix considers Monte Carlo methods for computing FIMs when closed forms are not attainable.

研究动机与目标

• 评估并比较期望FIM与观测FIM在估计最大似然估计协方差方面的性能。
• 评估基于期望FIM或观测FIM的估计器在有限样本和渐近设定下，哪个能产生更低的均方误差。
• 挑战广泛持有的观点，即观测FIM在MLE协方差估计中更优。
• 为在标准MLE正则性条件下，期望FIM的优越性能提供理论依据和数值验证。
• 探讨当闭式表达式不可用时，FIM的实用计算方法。

提出的方法

• 基于MLE渐近分布理论和二阶随机展开的理论分析。
• 推导观测FIM逆矩阵与期望FIM逆矩阵估计器在元素层面的均方误差（MSE）。
• 利用在正则性条件下观测FIM与期望FIM的渐近等价性，比较MSE性能。
• 在三个不同统计模型上进行数值实验，以验证理论发现。
• 在无闭式FIM表达式的模型中，应用蒙特卡洛方法进行FIM计算。
• 在SPSA中引入最优扰动分布，以实现随机逼近，与FIM估计相关。

实验结果

研究问题

• RQ1 在估计MLE协方差矩阵时，期望FIM的逆矩阵是否具有比观测FIM的逆矩阵更低的均方误差？
• RQ2 在何种条件下，期望FIM在MLE协方差估计的MSE方面优于观测FIM？
• RQ3 在有限样本和渐近情形下，期望FIM与观测FIM的性能如何比较？
• RQ4 在标准MLE正则性条件下，能否建立两种估计器之间MSE比较的理论？
• RQ5 在实际统计推断中，使用期望FIM替代观测FIM有何影响？

主要发现

• 在渐近情形下，期望FIM的逆矩阵在元素层面上的均方误差不大于观测FIM的逆矩阵。
• 在MLE的标准正则性条件下，包括似然函数的光滑性和可辨识性，期望FIM的优越性成立。
• 在三个不同模型上的数值研究证实了理论预测：期望FIM在协方差估计中产生更低的MSE。
• 该结果与广泛接受的观点相矛盾，即观测FIM因可直接从数据计算而更优。
• 即使在MLE处评估时，期望FIM也被证明是真实MLE协方差矩阵更准确的估计器，依据MSE准则。
• 本研究为重新考虑在MLE标准误估计中使用期望FIM提供了理论和实证依据。