[Paper Review] An asymptotic approximation of the marginal likelihood for general Markov models

Piotr Zwiernik|arXiv (Cornell University)|Dec 3, 2010

Bayesian Modeling and Causal Inference24 references3 citations

TL;DR

This paper derives an asymptotic approximation of the marginal likelihood for general Markov models on rooted binary trees with hidden variables, using real log-canonical thresholds to generalize the BIC score beyond regular models. The key result shows that the log-N penalty in the BIC approximation is determined by the number of leaves, degree-two inner nodes, and degree-three inner nodes in the tree, with a correction term when the root is a non-degenerate inner node.

ABSTRACT

The standard Bayesian Information Criterion (BIC) is derived under regularity conditions which are not always satisfied by the graphical models with hidden variables. In this paper we derive the BIC score for Bayesian networks in the case of binary data and when the underlying graph is a rooted tree and all the inner nodes represent hidden variables. This provides a direct generalization of a similar formula given by Rusakov and Geiger for naive Bayes models. The main tool used in this paper is a connection between asymptotic approximation of Laplace integrals and the real log-canonical threshold.

Motivation & Objective

• To extend the Bayesian Information Criterion (BIC) to singular models with hidden variables, particularly general Markov models on trees.
• To address the failure of standard BIC in models where the likelihood maximizer is not a regular point, such as in tree-structured Bayesian networks with hidden nodes.
• To establish a precise asymptotic approximation of the marginal likelihood using real log-canonical thresholds and Laplace integral asymptotics.
• To characterize how the structure of the tree—specifically the number of leaves, degree-two, and degree-three inner nodes—affects the marginal likelihood penalty term.
• To derive conditions under which an additional log log N term appears in the asymptotic expansion, depending on the root node's degeneracy.

Proposed method

• Uses asymptotic Laplace integral approximation to analyze the marginal likelihood in singular statistical models.
• Applies the theory of real log-canonical thresholds (RLCT) to characterize the asymptotic behavior of the marginal likelihood when the likelihood maximizer is singular.
• Models the general Markov process on a rooted binary tree with all internal nodes unobserved, focusing on the distribution over observed leaf variables.
• Defines degenerate inner nodes as those where all pairwise covariances among leaves separated by the node vanish, and uses this to classify tree structures.
• Derives the RLCT of the model ideal using decomposition into subtrees and applies results from algebraic geometry and singularity theory.
• Uses the formula $ Z(N) = ilde{ heta}_N - rac{1}{4}(3n + l_2 + 5l_3) ext{log}N + O(1) $, with correction when the root is non-degenerate, and includes a $ (m-1) ext{log log}N $ term when multiplicity exceeds 1.

Experimental results

Research questions

• RQ1 How does the marginal likelihood behave asymptotically in general Markov models with hidden variables on a rooted tree?
• RQ2 What is the correct form of the BIC approximation when the likelihood maximizer is singular, such as in tree models with hidden nodes?
• RQ3 How do the number and types of inner nodes (degree 2, 3) in the tree affect the log-N penalty in the marginal likelihood approximation?
• RQ4 Under what conditions does an additional $ ext{log log}N $ term appear in the asymptotic expansion of the marginal likelihood?
• RQ5 Can the structure of the observed sample covariance matrix (specifically zero entries) fully determine the asymptotic behavior of the marginal likelihood in such models?

Key findings

• The asymptotic marginal likelihood is given by $ Z(N) = ilde{ heta}_N - rac{1}{4}(3n + l_2 + 5l_3) ext{log}N + O(1) $, where $ n $ is the number of leaves, $ l_2 $ the number of degree-two inner nodes, and $ l_3 $ the number of degree-three inner nodes.
• When the root node is a non-degenerate inner node, the penalty term becomes $ rac{1}{4}(3n + l_2 + 5l_3 - 1) ext{log}N $, reflecting a structural correction.
• If the root is degenerate and all neighbors are degenerate, the multiplicity of the singularity is 1, and no additional $ ext{log log}N $ term appears.
• When the multiplicity of the singularity is greater than 1, the asymptotic expansion includes an additional $ (m-1) ext{log log}N $ term with $ m eq 1 $, affecting model selection.
• The sample covariance matrix's zero entries completely determine the asymptotic structure of the marginal likelihood, particularly through the identification of degenerate nodes.
• The real log-canonical threshold (RLCT) of the model is $ rac{1}{4}(3n + l_2 + 5l_3) $, and the multiplicity is 1 unless the root and its neighbors are degenerate, in which case it may be greater than 1.