[Paper Review] Pairwise MRF Calibration by Perturbation of the Bethe Reference Point

Cyril Furtlehner, Yufei Han+2 more|arXiv (Cornell University)|Oct 19, 2012

Bayesian Modeling and Causal Inference43 references3 citations

TL;DR

This paper proposes a novel framework for solving the inverse Ising and Gaussian Markov random field (MRF) problems by perturbing a Bethe mean-field reference point derived from a maximum spanning tree of pairwise mutual information. It introduces three methods—iterative proportional scaling, natural gradient descent in a reduced space, and dual weight propagation on Bethe-dual graphs—enabling efficient, BP-compatible sparse model selection with exact linear response computation in O(N²) time for a subclass of planar models.

ABSTRACT

We investigate different ways of generating approximate solutions to the pairwise Markov random field (MRF) selection problem. We focus mainly on the inverse Ising problem, but discuss also the somewhat related inverse Gaussian problem because both types of MRF are suitable for inference tasks with the belief propagation algorithm (BP) under certain conditions. Our approach consists in to take a Bethe mean-field solution obtained with a maximum spanning tree (MST) of pairwise mutual information, referred to as the \emph{Bethe reference point}, for further perturbation procedures. We consider three different ways following this idea: in the first one, we select and calibrate iteratively the optimal links to be added starting from the Bethe reference point; the second one is based on the observation that the natural gradient can be computed analytically at the Bethe point; in the third one, assuming no local field and using low temperature expansion we develop a dual loop joint model based on a well chosen fundamental cycle basis. We indeed identify a subclass of planar models, which we refer to as \emph{Bethe-dual graph models}, having possibly many loops, but characterized by a singly connected dual factor graph, for which the partition function and the linear response can be computed exactly in respectively O(N) and $O(N^2)$ operations, thanks to a dual weight propagation (DWP) message passing procedure that we set up. When restricted to this subclass of models, the inverse Ising problem being convex, becomes tractable at any temperature. Experimental tests on various datasets with refined $L_0$ or $L_1$ regularization procedures indicate that these approaches may be competitive and useful alternatives to existing ones.

Motivation & Objective

• Address the challenge of selecting sparse, BP-compatible MRF structures for large-scale real-time inference.
• Overcome limitations of existing inverse Ising and Gaussian MRF methods that lack compatibility with belief propagation or fail under high correlation.
• Develop theoretically grounded, efficient alternatives to heuristic or computationally expensive approaches for model selection in pairwise MRFs.
• Enable tractable inverse Ising problem solutions at any temperature by identifying a subclass of planar models with singly connected dual factor graphs.
• Integrate sparsity constraints with walk-summability conditions to ensure stable and efficient BP inference in high-dimensional settings.

Proposed method

• Use the Bethe approximation based on a maximum spanning tree (MST) of pairwise mutual information as a reference point for perturbation.
• Apply iterative proportional scaling (IPS) to iteratively add optimal links to the MST backbone, ensuring BP compatibility and sparsity.
• Leverage analytical natural gradient computation at the Bethe point to guide optimization in a reduced parameter space.
• Develop dual weight propagation (DWP) for Bethe-dual graphs, enabling exact partition function and linear response computation in O(N) and O(N²) time, respectively.
• Use low-temperature expansion to derive a dual-loop joint model based on a fundamental cycle basis, enabling exact inference in the Bethe-dual graph subclass.
• Integrate L₀ or L₁ regularization into the model selection pipeline to enforce sparsity while maintaining walk-summability and BP compatibility.

Experimental results

Research questions

• RQ1 Can a Bethe mean-field solution derived from an MST of mutual information serve as a stable and informative reference point for inverse MRF model selection?
• RQ2 How can perturbation methods be systematically applied to the Bethe reference point to improve model likelihood and sparsity while preserving BP compatibility?
• RQ3 What is the computational and theoretical advantage of using dual weight propagation (DWP) on Bethe-dual graphs for exact linear response and partition function computation?
• RQ4 In what conditions does the inverse Ising problem become tractable despite the presence of loops, and can this be generalized to non-planar or high-correlation settings?
• RQ5 How do the proposed methods compare to existing approaches in terms of predictive performance, sparsity, and computational efficiency on real-world datasets like Sioux-Falls and IAU traffic data?

Key findings

• The Bethe-dual graph subclass enables exact computation of the partition function in O(N) and linear response in O(N²) time using dual weight propagation (DWP), significantly improving efficiency.
• The inverse Ising problem becomes tractable at any temperature when restricted to Bethe-dual graphs due to convexity and exact inference capabilities.
• Iterative proportional scaling from the Bethe reference point produces sparse, BP-compatible models that outperform baseline methods on Gaussian and Ising data, especially under high correlation.
• On the Sioux-Falls traffic dataset, the sparse Gaussian model with greedy selection matched the performance of the full Gaussian model and outperformed k-NN, while the best Ising model used type (ii) encoding with BP.
• On the IAU dataset, the full Gaussian model was competitive with k-NN, but the best Ising model performed poorly due to high correlations, indicating limitations in modeling multi-modal or highly correlated systems.
• The method based on natural gradient descent at the Bethe point enables efficient optimization with precise susceptibility evaluation, overcoming prior computational bottlenecks in mean-field approaches.