[Paper Review] AIMS:Average Information Matrix Splitting

Shengxin Zhu, Tong-Xiang Gu+1 more|arXiv (Cornell University)|May 24, 2016

Statistical Mechanics and Entropy18 references4 citations

TL;DR

This paper introduces AIMS (Average Information Matrix Splitting), a computational method for linear mixed models that splits the average of the observed and Fisher information matrices into a computationally simple essential part and a negligible random zero matrix. The key contribution is proving that even when the variance-covariance matrix is not linearly dependent on variance components, the essential part of the average information matrix remains analytically tractable and computationally efficient, significantly reducing the cost of variance parameter estimation in large-scale models such as those in genome-wide association studies.

ABSTRACT

For linear mixed models with co-variance matrices which are not linearly dependent on variance component parameters, we prove that the average of the observed information and the Fisher information can be split into two parts. The essential part enjoys a simple and computational friendly formula, while the other part which involves a lot of computations is a random zero matrix and thus is negligible.

Motivation & Objective

• To address the high computational cost of evaluating the observed information matrix in maximum likelihood estimation for large-scale linear mixed models.
• To extend the average information matrix splitting technique beyond models with linearly dependent covariance matrices.
• To provide a computationally efficient alternative to the Fisher-scoring algorithm by simplifying the Hessian approximation.
• To enable scalable inference in high-throughput statistical applications such as genome-wide association studies.

Proposed method

• Proposes splitting the average of the observed and Fisher information matrices into two components: an essential part and a remainder.
• Derives analytical expressions for the essential part of the average information matrix using the residual log-likelihood and projection matrix P.
• Shows that the remainder part is a random zero matrix in expectation, making it negligible for estimation.
• Uses the identity P = H⁻¹ − H⁻¹X(XᵀH⁻¹X)⁻¹XᵀH⁻¹ to simplify trace and quadratic form computations.
• Applies the law of iterated expectations and matrix trace identities to prove that the expected value of the remainder is zero.
• Establishes that the essential part of the average information matrix matches the Fisher information matrix in expectation, ensuring statistical consistency.

Experimental results

Research questions

• RQ1 Can the average information matrix be effectively split into a computationally simple part and a negligible remainder in linear mixed models with non-linearly dependent covariance matrices?
• RQ2 Does the remainder part of the average information matrix vanish in expectation, even when the variance-covariance matrix is not linear in the parameters?
• RQ3 How can the computational burden of Hessian evaluation in maximum likelihood estimation be reduced without sacrificing statistical accuracy?
• RQ4 Is the essential part of the average information matrix sufficient for efficient variance parameter estimation in large-scale models?
• RQ5 Can the AIMS framework be applied beyond linearly parameterized covariance structures, such as in complex genetic or spatial models?

Key findings

• The average information matrix can be decomposed into an essential part with a simple analytical formula and a remainder that is a random zero matrix in expectation.
• The essential part of the average information matrix is computationally efficient and avoids expensive Hessian evaluations.
• The remainder part has zero expectation, making it negligible for estimation purposes, even when the covariance matrix is not linearly dependent on variance parameters.
• The essential part matches the Fisher information matrix in expectation, preserving statistical efficiency.
• The method significantly reduces computational cost in maximum likelihood estimation for large-scale linear mixed models.
• The approach enables scalable inference in high-dimensional settings such as genome-wide association studies.