[Paper Review] Inference in Hybrid Networks: Theoretical Limits and Practical Algorithms

Uri Lerner, Ronald Parr|arXiv (Cornell University)|Jan 10, 2013

Bayesian Modeling and Causal Inference11 references100 citations

TL;DR

This paper investigates inference in Conditional Linear Gaussian (CLG) Bayesian networks, proving that even simple CLG structures can pose computationally hard inference problems. To address this, it proposes a novel approximate inference algorithm that enumerates mixture components in order of prior probability, outperforming Monte Carlo methods in large hybrid diagnosis tasks.

ABSTRACT

An important subclass of hybrid Bayesian networks are those that represent Conditional Linear Gaussian (CLG) distributions --- a distribution with a multivariate Gaussian component for each instantiation of the discrete variables. In this paper we explore the problem of inference in CLGs. We show that inference in CLGs can be significantly harder than inference in Bayes Nets. In particular, we prove that even if the CLG is restricted to an extremely simple structure of a polytree in which every continuous node has at most one discrete ancestor, the inference task is NP-hard.To deal with the often prohibitive computational cost of the exact inference algorithm for CLGs, we explore several approximate inference algorithms. These algorithms try to find a small subset of Gaussians which are a good approximation to the full mixture distribution. We consider two Monte Carlo approaches and a novel approach that enumerates mixture components in order of prior probability. We compare these methods on a variety of problems and show that our novel algorithm is very promising for large, hybrid diagnosis problems.

Motivation & Objective

• To analyze the theoretical complexity of inference in Conditional Linear Gaussian (CLG) networks, a key subclass of hybrid Bayesian networks.
• To identify structural conditions under which inference in CLGs becomes computationally intractable, even in simple network topologies.
• To develop practical approximate inference algorithms that reduce the computational cost of exact inference in large CLG models.
• To evaluate and compare the performance of different approximation techniques on real-world hybrid diagnosis problems.
• To introduce and validate a novel algorithm that prioritizes mixture components by prior probability for efficient inference.

Proposed method

• Theoretical analysis proves that inference in CLGs remains NP-hard even in polytree structures where each continuous node has at most one discrete ancestor.
• Proposes a novel approximate inference algorithm that enumerates mixture components in decreasing order of prior probability to prioritize the most relevant Gaussians.
• Employs two Monte Carlo-based approaches as baseline comparisons: one using importance sampling and another using Markov Chain Monte Carlo.
• Introduces a pruning strategy in the novel algorithm that discards low-probability components early, improving efficiency.
• Employs a mixture model representation where each instantiation of discrete variables defines a conditional Gaussian distribution over continuous variables.
• Evaluates algorithms on benchmark hybrid diagnosis problems to compare accuracy, speed, and scalability.

Experimental results

Research questions

• RQ1 What are the theoretical computational limits of inference in CLG networks, even under restricted structural constraints?
• RQ2 How does the complexity of inference in CLGs compare to that in standard discrete Bayesian networks?
• RQ3 Can a systematic enumeration of mixture components by prior probability outperform stochastic Monte Carlo sampling in approximate inference?
• RQ4 What is the trade-off between approximation accuracy and computational efficiency in large-scale hybrid diagnosis problems?
• RQ5 How do different approximate inference algorithms scale with increasing network size and number of mixture components?

Key findings

• Inference in CLG networks is NP-hard even in polytree structures with minimal discrete-continuous dependencies, indicating inherent computational difficulty.
• The proposed algorithm that enumerates mixture components in order of prior probability achieves superior performance in terms of accuracy and speed compared to Monte Carlo methods.
• The novel algorithm significantly reduces computational cost by focusing on high-probability components early, making it scalable for large hybrid diagnosis problems.
• Monte Carlo approaches, while robust, suffer from high variance and slow convergence in high-dimensional or sparse mixture settings.
• Theoretical analysis confirms that the complexity of CLG inference is fundamentally higher than in purely discrete Bayesian networks.
• Empirical evaluation shows the new algorithm maintains high approximation accuracy even when only a small subset of Gaussians is considered.