[Paper Review] Estimating Continuous Distributions in Bayesian Classifiers

George H. John, Pat Langley|arXiv (Cornell University)|Feb 20, 2013

Bayesian Modeling and Causal InferenceComputer Science21 references2,918 citations

TL;DR

The paper investigates nonparametric density estimation for continuous variables in Bayesian classifiers, showing kernel density estimation can outperform Gaussian assumptions. It provides experimental results across various domains demonstrating reduced error.

ABSTRACT

When modeling a probability distribution with a Bayesian network, we are faced with the problem of how to handle continuous variables. Most previous work has either solved the problem by discretizing, or assumed that the data are generated by a single Gaussian. In this paper we abandon the normality assumption and instead use statistical methods for nonparametric density estimation. For a naive Bayesian classifier, we present experimental results on a variety of natural and artificial domains, comparing two methods of density estimation: assuming normality and modeling each conditional distribution with a single Gaussian; and using nonparametric kernel density estimation. We observe large reductions in error on several natural and artificial data sets, which suggests that kernel estimation is a useful tool for learning Bayesian models.

Motivation & Objective

• Motivate the challenge of modeling continuous variables in Bayesian networks without assuming normality.
• Assess nonparametric density estimation as an alternative to discretization or single Gaussian models.
• Evaluate the impact of kernel density estimation on classifier accuracy across diverse domains.

Proposed method

• Compare two density estimation approaches for conditional distributions: (i) assuming normality with a single Gaussian per conditional, (ii) nonparametric kernel density estimation.
• Apply these methods within a naive Bayesian classifier framework.
• Experimentally evaluate on natural and artificial data sets.
• Analyze and compare classification error reductions between the two approaches.

Experimental results

Research questions

• RQ1 Does kernel density estimation improve classification accuracy over Gaussian assumptions in Bayesian classifiers?
• RQ2 How does nonparametric density estimation perform across different natural and artificial domains?
• RQ3 What are the practical benefits of kernel methods for learning Bayesian models with continuous variables?

Key findings

• Kernel density estimation yields large reductions in error on several data sets.
• Nonparametric density estimation can outperform the single Gaussian approach in Bayesian classifiers.
• Results suggest kernel-based learning is a useful tool for modeling continuous distributions in Bayesian networks.