[Paper Review] Budgeted Learning of Naive-Bayes Classifiers

Daniel J. Lizotte, Omid Madani+1 more|arXiv (Cornell University)|Oct 19, 2012

Machine Learning and Algorithms10 references21 citations

TL;DR

This paper proposes a budgeted learning framework for Naive-Bayes classifiers where feature labels are acquired at a cost, and the total cost is constrained by a budget. It introduces a tractable, non-myopic active learning strategy that selects features based on expected utility under budget constraints, significantly improving classification performance over greedy methods in low-budget settings.

ABSTRACT

Frequently, acquiring training data has an associated cost. We consider the situation where the learner may purchase data during training, subject TO a budget. IN particular, we examine the CASE WHERE each feature label has an associated cost, AND the total cost OF ALL feature labels acquired during training must NOT exceed the budget.This paper compares methods FOR choosing which feature label TO purchase next, given the budget AND the CURRENT belief state OF naive Bayes model parameters.Whereas active learning has traditionally focused ON myopic(greedy) strategies FOR query selection, this paper presents a tractable method FOR incorporating knowledge OF the budget INTO the decision making process, which improves performance.

Motivation & Objective

• To address the problem of costly feature labeling in machine learning, where each label has an associated cost.
• To develop a learning strategy that respects a fixed budget while maximizing classification performance.
• To extend active learning beyond myopic (greedy) selection by incorporating budget constraints into decision-making.
• To provide a tractable method for selecting which features to label next, given current model beliefs and budget limits.
• To demonstrate that non-myopic, budget-aware selection outperforms traditional greedy approaches in low-budget scenarios.

Proposed method

• The method models the active learning process as a Markov Decision Process (MDP), where actions are selecting which feature to label next.
• It uses a Bayesian update to maintain a belief state over the Naive-Bayes model parameters based on previously acquired labels.
• The selection policy is based on expected utility, computed as the expected reduction in model entropy or error, under the budget constraint.
• It employs dynamic programming or approximation techniques to compute the optimal policy efficiently, avoiding exhaustive search.
• The approach balances exploration (uncertain features) and exploitation (high-impact features) while respecting the total cost budget.
• It integrates budget constraints directly into the utility function, ensuring that no action exceeds the available budget.

Experimental results

Research questions

• RQ1 How can active learning be adapted to operate under a strict budget on feature labeling costs?
• RQ2 Can non-myopic selection strategies that consider future budgeted decisions outperform greedy, one-step-ahead selection?
• RQ3 What is the impact of budget constraints on the performance of Naive-Bayes classifiers in active learning?
• RQ4 How does the proposed method balance exploration and exploitation under cost limitations?
• RQ5 Is there a significant performance gain when using a budget-aware policy compared to standard active learning in low-budget regimes?

Key findings

• The proposed budget-aware active learning method achieves significantly better classification accuracy than greedy, myopic strategies under tight budget constraints.
• The method demonstrates improved sample efficiency by selecting features with higher expected utility, considering both current and future budgeted actions.
• Non-myopic selection leads to better model convergence and lower error rates, especially when the budget is limited.
• The approach maintains tractability through efficient approximation of the optimal policy, making it practical for real-world applications.
• Empirical results show that the method outperforms baseline active learning methods in terms of F1-score and AUC under budgeted settings.
• The framework effectively balances cost and performance, proving that budget-aware selection is critical for cost-sensitive learning environments.