[Paper Review] Naive Feature Selection: a Nearly Tight Convex Relaxation for Sparse Naive Bayes

Armin Askari, Alexandre d’Aspremont+1 more|arXiv (Cornell University)|May 23, 2019

Rough Sets and Fuzzy Logic4 citations

TL;DR

This paper proposes a sparse variant of naive Bayes that directly enforces sparsity in the classification coefficients via a convex relaxation, enabling exact or near-tight solutions for feature selection. The method achieves state-of-the-art statistical performance comparable to LASSO and recursive feature elimination, but with near-linear computational complexity, training in under 15 seconds on a 1.6M-point, 12M-feature text dataset using a non-optimized CPU implementation.

ABSTRACT

Due to its linear complexity, naive Bayes classification remains an attractive supervised learning method, especially in very large-scale settings. We propose a sparse version of naive Bayes, which can be used for feature selection. This leads to a combinatorial maximum-likelihood problem, for which we provide an exact solution in the case of binary data, or a bound in the multinomial case. We prove that our convex relaxation bounds becomes tight as the marginal contribution of additional features decreases, using a priori duality gap bounds dervied from the Shapley-Folkman theorem. We show how to produce primal solutions satisfying these bounds. Both binary and multinomial sparse models are solvable in time almost linear in problem size, representing a very small extra relative cost compared to the classical naive Bayes. Numerical experiments on text data show that the naive Bayes feature selection method is as statistically effective as state-of-the-art feature selection methods such as recursive feature elimination, $l_1$-penalized logistic regression and LASSO, while being orders of magnitude faster.

Motivation & Objective

• To develop a sparse naive Bayes model that directly controls feature selection cardinality without hyperparameter tuning.
• To provide an exact solution for binary (Bernoulli) data and a tight convex relaxation for multinomial data.
• To achieve training complexity nearly linear in data size, preserving the scalability of classical naive Bayes.
• To demonstrate that the method matches or exceeds the statistical performance of state-of-the-art feature selection techniques while being orders of magnitude faster.
• To show that the relaxation becomes increasingly tight as the marginal contribution of features diminishes.

Proposed method

• Formulates a combinatorial maximum-likelihood problem with explicit cardinality constraints on the classification coefficient vector.
• Derives an exact solution for binary (Bernoulli) data using duality and KKT conditions, reducing to a thresholding operation on feature coefficients.
• Proposes a convex relaxation for multinomial data using a Lagrangian dual approach, yielding an upper bound on the optimal objective.
• Employs a two-stage algorithm: first computes feature coefficients via standard naive Bayes, then applies a thresholding rule to enforce sparsity.
• Uses a duality-based approach to show that the relaxation is tight when marginal feature contributions are small.
• Leverages the structure of the naive Bayes classifier to ensure the final model remains interpretable and computationally efficient.

Experimental results

Research questions

• RQ1 Can a convex relaxation be derived for sparse naive Bayes that is both computationally efficient and statistically tight?
• RQ2 Does the proposed method achieve exact sparsity control without requiring hyperparameter tuning for sparsity level?
• RQ3 How does the performance of the sparse naive Bayes model compare to LASSO, recursive feature elimination, and other feature selection baselines in terms of accuracy and speed?
• RQ4 Under what conditions does the convex relaxation become tight, particularly in relation to feature marginal contributions?
• RQ5 Can the method scale to very large datasets—such as those with millions of features and training points—while maintaining near-linear training time?

Key findings

• The proposed sparse naive Bayes model achieves statistical performance comparable to state-of-the-art methods such as LASSO and recursive feature elimination on text classification tasks.
• For a large-scale dataset with 1.6 million training points and approximately 12 million features, the model trains in less than 15 seconds using a non-optimized CPU implementation.
• The method provides an exact solution for binary (Bernoulli) data and a convex relaxation that becomes increasingly tight as the marginal contribution of features decreases.
• The model outperforms simple baselines like thresholded naive Bayes and odds ratio feature selection in both accuracy and efficiency.
• The algorithm scales almost linearly with the number of features and data points, matching the complexity of classical naive Bayes while adding minimal overhead.
• The approach eliminates the need for hyperparameter tuning to achieve a desired sparsity level, as sparsity is directly controlled via the cardinality constraint.