[Paper Review] Naive Bayes and Text Classification I - Introduction and Theory

Sebastian Raschka|arXiv (Cornell University)|Jan 1, 2014

Text and Document Classification Technologies4 references81 citations

TL;DR

This paper provides a comprehensive theoretical foundation for Naive Bayes classification in text mining, explaining how Bayes' theorem enables efficient probabilistic classification under the assumption of feature independence. It details the application of Multinomial and Multi-variate Bernoulli Naive Bayes models to text, including term frequency, TF-IDF weighting, and Laplace smoothing, demonstrating strong performance in spam filtering and document classification tasks.

ABSTRACT

Naive Bayes classifiers, a family of classifiers that are based on the popular Bayes' probability theorem, are known for creating simple yet well performing models, especially in the fields of document classification and disease prediction. In this article, we will look at the main concepts of naive Bayes classification in the context of document categorization.

Motivation & Objective

• To provide a clear theoretical introduction to Naive Bayes classification using Bayes' theorem.
• To explain the mathematical foundations of class-conditional, prior, and posterior probabilities in the context of text classification.
• To compare Multinomial and Multi-variate Bernoulli Naive Bayes models for text data.
• To demonstrate the use of term frequency, TF-IDF, and additive smoothing in improving model robustness.
• To guide researchers through the design and implementation of a Naive Bayes classifier for text using real-world SMS data.

Proposed method

• Applies Bayes' theorem to compute posterior probabilities for class assignment: P(ωj | xi) = P(xi | ωj)P(ωj)/P(xi).
• Uses maximum-likelihood estimation for class-conditional probabilities under the naive independence assumption: P(x | ωj) = ∏P(xk | ωj).
• Employs Laplace smoothing (additive smoothing) to handle zero-frequency problems: P(xi | ωj) = (Nxi,ωj + α)/(Nωj + α·V).
• Utilizes the bag-of-words model with tokenization, stop-word removal, stemming, and n-grams for feature extraction.
• Applies TF-IDF weighting to emphasize rare, informative terms: Tf-idf = tfn(t,d) · log(nd/nd(t)).
• Compares Multinomial Naive Bayes (based on term counts) and Multi-variate Bernoulli Naive Bayes (based on binary presence/absence) for text classification.

Experimental results

Research questions

• RQ1 How does the Naive Bayes classifier compute the posterior probability of a document belonging to a given class?
• RQ2 What is the impact of additive smoothing on handling unseen words in text classification?
• RQ3 How do Multinomial and Multi-variate Bernoulli Naive Bayes models differ in their treatment of term frequency and document representation?
• RQ4 In what scenarios does the multinomial model outperform the multi-variate Bernoulli model for text classification?
• RQ5 How do feature engineering choices like stop-word removal and stemming affect the performance of Naive Bayes classifiers?

Key findings

• The multinomial Naive Bayes model generally outperforms the multi-variate Bernoulli model on large-vocabulary text datasets.
• Additive smoothing (e.g., Laplace smoothing with α=1) effectively prevents zero probabilities for unseen feature values.
• TF-IDF weighting improves model performance by downweighting common, less discriminative terms such as stop words.
• The multinomial model, which uses raw or normalized term frequencies, is more suitable for text where word count carries semantic weight.
• Naive Bayes classifiers remain effective despite the violation of the conditional independence assumption, especially with small to moderate datasets.
• The model's performance is highly sensitive to preprocessing steps such as stop-word removal, stemming, and n-gram selection, which should be systematically evaluated.