[Paper Review] Information-based inference for singular models and finite sample sizes: A frequentist information criterion

Colin LaMont, Paul A. Wiggins|arXiv (Cornell University)|Jun 19, 2015

Statistical Methods and Inference30 references3 citations

TL;DR

This paper introduces the Frequentist Information Criterion (QIC), a new information criterion that improves model selection for singular models and finite sample sizes by replacing the AIC's asymptotic approximation of model complexity with a frequentist approximation that adapts to likelihood geometry, sample size, and parameter manifold structure. QIC significantly reduces bias in complexity estimation, outperforming AIC and BIC in both regular and singular models, especially in small samples or with sloppy/identifiable parameter spaces.

ABSTRACT

In the information-based paradigm of inference, model selection is performed by selecting the candidate model with the best estimated predictive performance. The success of this approach depends on the accuracy of the estimate of the predictive complexity. In the large-sample-size limit of a regular model, the predictive performance is well estimated by the Akaike Information Criterion (AIC). However, this approximation can either significantly under or over-estimating the complexity in a wide range of important applications where models are either non-regular or finite-sample-size corrections are significant. We introduce an improved approximation for the complexity that is used to define a new information criterion: the Frequentist Information Criterion (QIC). QIC extends the applicability of information-based inference to the finite-sample-size regime of regular models and to singular models. We demonstrate the power and the comparative advantage of QIC in a number of example analyses.

Motivation & Objective

• To address the failure of AIC in singular models and finite-sample regimes due to biased complexity estimation.
• To identify the root causes of AIC's bias, including model singularity, finite sample size, and parameter unidentifiability.
• To develop a new complexity approximation—frequentist complexity—that adapts to likelihood structure, sample size, and training algorithm.
• To propose QIC as a robust, improved information criterion that outperforms AIC and BIC in predictive performance for both regular and singular models.
• To demonstrate QIC's superiority through empirical analyses on biophysical and cell biology data.

Proposed method

• Propose a new approximation for model complexity called 'frequentist complexity', based on the estimated model at the MLE parameters rather than a universal function of dimension and sample size.
• Define the Frequentist Information Criterion (QIC) as the sum of the negative log-likelihood and the frequentist complexity, with the complexity derived from the curvature and geometry of the parameter manifold.
• Use the Laplace approximation to justify the asymptotic equivalence of QIC and AIC in the large-sample-size limit for regular models.
• Account for model singularity by analyzing how unidentifiable parameters and extrinsic curvature affect the variance of the MLE, leading to bias in complexity estimation.
• Apply QIC to real data examples, including Fourier-based models of neutrino intensity and resonance models, to validate performance.
• Demonstrate that QIC adapts to model structure—yielding larger values for high-multiplicity models and smaller values for sloppy models—compared to AIC.

Experimental results

Research questions

• RQ1 How does AIC fail in singular models and finite-sample regimes due to inaccurate complexity estimation?
• RQ2 What role do parameter unidentifiability, sample size, and parameter manifold geometry play in determining model complexity?
• RQ3 Can a frequentist complexity approximation be derived that adapts to the likelihood function, training algorithm, and sample size, improving on AIC’s fixed-dimension assumption?
• RQ4 How does QIC compare to AIC and BIC in predictive performance across regular and singular models?
• RQ5 In what scenarios does QIC significantly outperform AIC, particularly in small samples or with sloppy parameter spaces?

Key findings

• QIC is asymptotically equivalent to AIC for regular models in the large-sample-size limit, ensuring consistency with established methods.
• For singular models, QIC can be substantially larger or smaller than AIC depending on model structure—e.g., much larger for high-multiplicity models, much smaller for sloppy models—due to its adaptive complexity estimation.
• The frequentist complexity in QIC reduces bias in complexity estimation compared to AIC, leading to improved predictive performance in finite samples.
• QIC outperforms AIC and BIC in model selection across multiple example analyses, particularly in cases where AIC’s assumptions break down.
• The presence or absence of extrinsic curvature in the parameter manifold significantly affects the accuracy of MLE variance estimation, which QIC accounts for via its geometry-aware complexity.
• In the Fourier-based model of neutrino intensity, QIC correctly captures the complexity of seasonal dependence, demonstrating robustness in real-world data with non-i.i.d. structure.